[00:00:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.02, 4.97, 3.70 [00:01:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.13, 5.46, 4.11 [00:01:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.57, 10.96, 9.02 [00:03:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.06, 11.25, 9.39 [00:04:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.37, 12.85, 10.39 [00:05:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.77, 3.73 [00:06:07] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 10621 MB (27% inode=96%); [00:06:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.96, 3.69 [00:08:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 11.52, 10.47 [00:09:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.79, 9.97, 9.50 [00:10:18] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 11106 MB (28% inode=96%); [00:11:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.28, 3.44 [00:12:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.51, 8.98, 9.68 [00:13:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.10, 3.34 [00:16:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.21, 3.38 [00:43:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.56, 1.94 [00:49:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.54, 1.31, 1.69 [00:57:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.73, 2.78, 2.07 [00:57:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.65, 2.93 [00:59:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.59, 2.88, 2.73 [01:03:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.96, 1.93 [01:07:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.38, 2.78, 2.20 [01:25:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.75, 1.99 [01:35:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 9.87, 8.55 [01:37:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.34, 1.44, 1.67 [01:37:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.95, 9.52, 8.58 [01:44:10] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 1.92, 1.81 [01:45:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.72, 2.00 [01:51:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 3.94, 2.73 [01:51:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 1.86, 1.95 [01:53:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.74, 1.89 [01:57:40] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.16, 2.11, 1.98 [01:59:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.84, 1.84, 1.88 [02:01:36] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.88, 2.06, 1.95 [02:05:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 4.00, 3.77 [02:09:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.54, 4.85, 4.03 [02:10:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 1.71, 2.00 [02:11:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.57, 3.04 [02:12:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.87, 2.03 [02:13:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.32, 3.01 [02:14:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.83, 1.99 [02:14:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.72, 3.75 [02:18:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.44, 2.62, 3.30 [02:20:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.18, 2.24, 2.06 [02:30:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.15, 1.69, 1.90 [02:31:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.56, 1.90 [02:35:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.49, 1.97, 1.98 [02:42:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.74, 1.22, 1.61 [02:48:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.49, 2.20, 1.86 [02:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.67, 3.11 [03:00:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.37, 3.07 [03:06:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.87, 9.60, 8.52 [03:08:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.76, 8.89, 8.40 [03:12:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.54, 1.93 [03:18:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.09, 1.89, 1.92 [03:20:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.90, 1.93 [03:30:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.08, 2.21, 1.95 [03:32:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.96, 1.88 [03:36:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.81, 2.21, 2.02 [03:38:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.86, 1.91 [03:40:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 1.97, 1.94 [03:40:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.25, 2.66 [03:42:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.22, 1.64, 1.82 [03:42:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 2.88, 2.60 [03:46:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.35, 1.95, 1.87 [03:47:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.74, 1.98 [03:52:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.81, 1.86 [03:56:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.38, 1.67 [04:01:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.81, 2.16, 1.98 [04:03:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.24, 1.88, 1.91 [04:09:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.90, 2.78, 2.22 [04:13:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.75, 1.79, 1.95 [04:15:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.31, 1.96, 1.99 [04:27:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.78, 1.92, 1.67 [04:29:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.73, 1.93 [04:33:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.73, 1.39, 1.54 [04:35:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [04:35:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.97, 1.95 [04:37:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.86, 1.91 [04:39:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.67, 2.15, 2.00 [04:44:08] PROBLEM - agentisai.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for agentisai.com could not be found [04:45:11] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw121 mw132 [04:49:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.87, 2.00 [04:51:20] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.78, 1.60 [04:51:43] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:51:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.74, 2.28, 2.14 [04:53:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.68, 1.66, 1.57 [04:54:14] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [04:54:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw121 mw122 mw132 [04:56:40] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is CRITICAL: connect to address vault.agentisai.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:01:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.30, 1.78, 1.97 [05:03:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.95, 2.01 [05:05:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.82, 1.95 [05:07:04] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw121 mw131 [05:09:19] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:09:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.80, 1.69 [05:11:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.47, 1.61, 1.63 [05:11:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.25, 2.09, 1.99 [05:13:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.76, 1.89 [05:15:13] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw121 mw131 mw132 mw142 [05:17:49] PROBLEM - vault.agentisai.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vault.agentisai.com could not be found [05:17:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.38, 1.97, 1.92 [05:18:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [05:18:34] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:19:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 2.00, 1.94 [05:20:39] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.078 second response time [05:20:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.68, 1.64 [05:20:47] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.48, 1.63, 1.62 [05:24:14] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:25:24] PROBLEM - agentisai.com - LetsEncrypt on sslhost is CRITICAL: connect to address agentisai.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:25:38] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [05:26:20] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.072 second response time [05:26:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.78, 1.72, 1.66 [05:27:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.35, 1.65 [05:28:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.68, 1.65, 1.63 [05:34:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.57, 2.54, 2.04 [05:35:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 1.71, 1.69 [05:36:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:37:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.58, 1.64 [05:38:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.89, 1.88 [05:39:03] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [05:42:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.35, 1.67 [05:42:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:43:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.67, 1.65 [05:44:58] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:45:07] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.67, 1.65, 1.64 [05:47:01] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [05:48:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.64, 1.69 [05:49:41] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:49:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.89, 1.74 [05:50:02] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [05:50:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.67, 1.62, 1.67 [05:51:40] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.034 second response time [05:51:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.61, 2.62, 2.03 [05:53:00] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.agentisai.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:55:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 2.75, 2.08 [05:56:57] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:57:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 2.62, 2.10 [05:57:32] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:58:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:58:20] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:59:28] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [05:59:40] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:56] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:00:58] PROBLEM - www.agentisai.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.agentisai.com could not be found [06:04:27] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:49] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.050 second response time [06:04:59] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [06:05:13] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:28] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:33] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:06:18] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:07:27] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [06:07:30] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [06:08:00] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:08:21] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.018 second response time [06:08:32] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.049 second response time [06:08:39] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:09:04] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:09:25] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.054 second response time [06:09:31] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:10:08] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.050 second response time [06:10:09] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [06:11:51] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:27] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 3.062 second response time [06:13:08] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:13:26] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:30] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.079 second response time [06:13:50] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.017 second response time [06:15:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [06:15:25] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.015 second response time [06:15:27] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:56] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 4 backends are down. mw121 mw122 mw132 mw141 [06:16:14] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.069 second response time [06:16:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.11, 2.66 [06:16:48] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:16:56] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:03] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:10] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:55] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.140 second response time [06:18:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.36, 3.03, 2.67 [06:18:53] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [06:18:57] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [06:19:23] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [06:19:23] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:20:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.73, 1.75, 1.41 [06:21:23] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.297 second response time [06:21:36] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.035 second response time [06:22:12] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:48] RECOVERY - agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [06:24:43] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.032 second response time [06:26:36] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:31] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.88, 1.64 [06:29:33] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.086 second response time [06:29:47] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:30:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.38, 1.67, 1.59 [06:31:02] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:31:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:31:52] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:31:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:31:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.58, 1.97 [06:33:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.089 second response time [06:33:41] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [06:33:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.97, 2.06, 2.09 [06:34:02] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.184 second response time [06:34:58] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.061 second response time [06:35:19] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:25] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:37:25] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:37:54] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.042 second response time [06:37:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.20, 1.73, 1.95 [06:39:02] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:39:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 2.08, 1.77 [06:39:46] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:40:25] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:40:46] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.040 second response time [06:41:01] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.92, 1.75 [06:41:29] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 22% [06:41:48] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:42:05] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:42:15] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [06:42:20] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.050 second response time [06:43:12] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.283 second response time [06:43:46] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.019 second response time [06:44:21] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:45:31] PROBLEM - vrcdev.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vrcdev.wiki could not be found [06:45:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 1.91, 1.91 [06:46:05] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:45] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.55, 1.64 [06:47:49] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.230 second response time [06:47:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.94, 1.92 [06:48:02] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:48:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:48:57] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 7.300 second response time [06:49:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 2.03, 1.96 [06:50:07] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:51:30] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:51:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.62, 1.94, 1.94 [06:52:17] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13150 MB (34% inode=96%); [06:52:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:56] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 2.042 second response time [06:53:06] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:18] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:24] PROBLEM - agentisai.com - LetsEncrypt on sslhost is CRITICAL: connect to address agentisai.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:53:58] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [06:54:40] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:55:13] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [06:55:27] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:55:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.71, 1.68 [06:55:49] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:40] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.016 second response time [06:57:30] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.327 second response time [06:57:30] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 1.90, 1.75 [06:57:48] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.031 second response time [06:57:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.42, 1.69 [06:58:16] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:58:25] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:41] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.065 second response time [06:59:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.31, 1.70, 1.70 [06:59:53] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:00:19] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.109 second response time [07:01:04] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:14] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:01:15] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.291 second response time [07:01:29] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:01:36] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.237 second response time [07:03:21] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [07:03:26] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13150 MB (34% inode=96%); [07:03:44] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [07:03:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.81, 1.80 [07:05:04] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:07] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:05:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.89, 2.44, 2.02 [07:06:09] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:06:30] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:04] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:07:36] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.033 second response time [07:07:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.89, 1.87 [07:08:24] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [07:08:28] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.045 second response time [07:08:49] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:00] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:01] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001897573471 secs [07:09:15] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:27] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:12] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:10:36] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:01] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.037 second response time [07:11:39] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:12:33] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [07:13:08] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.039 second response time [07:13:38] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:13:40] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:13:59] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:14:22] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.014 second response time [07:14:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:11] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:15] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:17] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.171 second response time [07:15:48] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.154 second response time [07:17:03] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [07:17:15] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.178 second response time [07:17:29] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [07:17:36] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:17:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.53, 1.69 [07:18:21] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.047 second response time [07:18:36] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:18:38] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:18:51] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.034 second response time [07:20:03] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [07:20:35] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13144 MB (34% inode=96%); [07:21:56] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [07:21:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.87, 2.06, 1.91 [07:22:06] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:22:31] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:22:33] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:37] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.017 second response time [07:22:39] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:22:55] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:23:39] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:42] PROBLEM - vrcdev.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address vrcdev.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:24:29] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [07:24:38] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:24:47] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:25:14] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:25:16] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:37] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:26:48] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [07:27:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 1.038 second response time [07:27:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.69, 1.83 [07:30:26] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:53] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.076 second response time [07:31:29] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [07:32:27] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:31] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:43] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.510 second response time [07:33:00] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 7.307 second response time [07:33:17] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.037 second response time [07:33:41] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.39, 1.67 [07:34:07] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:34:08] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:34:20] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.040 second response time [07:34:36] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 7.186 second response time [07:34:51] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.020 second response time [07:35:01] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:33] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:35:54] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [07:36:08] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.028 second response time [07:36:09] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.058 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:36:34] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:36:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.91, 2.35, 1.82 [07:37:00] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 2.043 second response time [07:37:31] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [07:37:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:37:47] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:37:59] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:38:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.96, 1.74 [07:39:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:57] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.62, 1.64 [07:40:41] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:52] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:54] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:38] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:04] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.002396643162 secs [07:42:12] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.017 second response time [07:43:04] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.309 second response time [07:43:21] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:43:37] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.198 second response time [07:43:38] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:39] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.016 second response time [07:44:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.91, 2.04, 1.82 [07:45:17] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:45:33] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.080 second response time [07:45:38] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.031 second response time [07:45:41] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.040 second response time [07:46:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.93, 1.81 [07:46:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:46:51] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:46:58] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:47:11] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:48:00] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.053 second response time [07:48:04] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:48:11] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:48:21] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:41] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [07:48:45] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.030 second response time [07:48:45] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.057 second response time [07:48:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.72, 2.60, 2.08 [07:50:02] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.029 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:50:11] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.035 second response time [07:50:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.55, 1.57, 1.68 [07:50:22] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:50:58] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [07:51:55] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:59] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:09] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:01] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13146 MB (34% inode=96%); [07:53:47] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:57] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.182 second response time [07:54:04] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.061 second response time [07:54:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.015 second response time [07:54:14] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:54:32] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:55:02] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:56:00] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [07:56:04] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.023 second response time [07:56:16] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [07:56:45] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [07:57:20] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:21] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:57:25] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:27] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:36] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [07:58:18] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:58:39] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.055 second response time [07:58:59] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:59:19] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35731 bytes in 3.076 second response time [07:59:21] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0003606975079 secs [07:59:22] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [07:59:26] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.021 second response time [08:00:02] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.038 second response time [08:00:54] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35731 bytes in 0.048 second response time [08:01:59] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:02:32] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [08:03:13] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:24] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 0.019 second response time [08:03:55] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13146 MB (34% inode=96%); [08:04:01] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:04:48] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:05] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:10] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 21% [08:05:55] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:06] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.025 second response time [08:06:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.55, 1.95 [08:06:59] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.29, 2.17, 1.87 [08:07:31] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [08:07:59] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:08:02] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:58] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.061 second response time [08:09:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 2.078 second response time [08:09:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:31] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:48] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 2.029 second response time [08:09:59] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:04] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:20] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:40] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:43] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.031 second response time [08:12:03] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [08:12:16] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [08:12:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 2.00, 1.91 [08:14:03] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.051 second response time [08:14:23] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:29] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:51] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 1.95, 1.89 [08:15:59] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.017 second response time [08:15:59] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:09] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:20] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:37] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [08:16:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.95, 1.83, 1.84 [08:17:28] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:17:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:18:24] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:18:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.91, 1.35, 1.69 [08:18:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.71, 1.91, 1.85 [08:19:07] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [08:20:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:20:39] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:21:43] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:19] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [08:22:26] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.075 second response time [08:22:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.22, 1.68, 1.79 [08:22:51] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:23:22] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [08:23:26] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:23:30] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:24:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.48, 1.66, 1.74 [08:24:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.43, 1.69 [08:25:04] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:25:24] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13137 MB (34% inode=96%); [08:25:34] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.046 second response time [08:25:50] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.500 second response time [08:26:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.46, 1.65 [08:27:10] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.139 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:27:12] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:27:20] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:06] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:43] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:51] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:15] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.243 second response time [08:29:19] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:30:42] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:10] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:26] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:32:24] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.027 second response time [08:32:37] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:54] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.393 second response time [08:32:55] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:33:23] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [08:34:34] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.054 second response time [08:34:41] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:46] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:06] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.068 second response time [08:36:46] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.015 second response time [08:36:51] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.028 second response time [08:36:52] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.069 second response time [08:37:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:18] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.227 second response time [08:39:31] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:39:32] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 1.027 second response time [08:40:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.86, 1.65 [08:41:10] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:12] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 2.15, 1.77 [08:43:08] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:17] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 2.028 second response time [08:43:26] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.68, 1.64 [08:44:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.48, 1.64, 1.61 [08:44:19] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:44:31] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 2.103 second response time [08:45:13] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:45:13] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [08:45:26] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 7.424 second response time [08:45:44] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:08] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:46:40] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:46:42] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:47:30] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.020 second response time [08:47:32] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:47:57] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [08:48:40] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:48:50] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.018 second response time [08:48:59] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:16] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:49:37] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.016 second response time [08:49:49] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:50:06] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:16] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:50:51] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:51:03] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:51:05] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0002776384354 secs [08:51:27] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:29] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [08:51:49] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.048 second response time [08:51:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:58] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:23] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.072 second response time [08:53:09] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.280 second response time [08:53:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 7.071 second response time [08:53:57] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:53:58] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:25] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 2.070 second response time [08:54:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.93, 1.79 [08:55:13] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.163 second response time [08:56:00] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.023 second response time [08:56:02] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [08:56:04] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 16% [08:56:12] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.93, 1.79 [08:56:48] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:57] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:57] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:58] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13140 MB (34% inode=96%); [08:58:10] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.032 second response time [08:58:16] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.061 second response time [08:58:34] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:14] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:59:24] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [08:59:33] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001350820065 secs [08:59:40] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:55] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:10] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.142 second response time [09:01:17] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:54] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.014 second response time [09:01:57] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.070 second response time [09:02:03] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:02:13] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:03:24] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:28] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [09:04:13] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 7.306 second response time [09:04:33] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.058 second response time [09:04:35] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:47] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:07] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.021 second response time [09:06:47] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:07:05] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.079 second response time [09:07:16] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 42% [09:08:06] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:08:16] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.083 second response time [09:08:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.32, 1.94, 1.98 [09:08:38] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.284 second response time [09:08:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:10:11] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:10:21] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.034 second response time [09:10:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.97, 3.76, 2.65 [09:10:48] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [09:11:02] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:11:38] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:11:47] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:11:54] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:12:09] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 28% [09:13:01] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.060 second response time [09:13:14] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.163 second response time [09:13:32] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:43] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 7.275 second response time [09:13:44] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:56] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 2.787 second response time [09:13:56] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.034 second response time [09:14:15] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:29] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:03] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:05] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:35] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:37] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:41] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:16:21] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.016 second response time [09:16:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:32] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13138 MB (34% inode=96%); [09:17:00] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:07] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:13] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:36] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.015 second response time [09:18:42] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:18:59] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:06] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 1.032 second response time [09:19:09] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:14] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:44] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.04, 0.01, 0.00 [09:19:46] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:04] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:04] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:58] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.063 second response time [09:21:18] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.053 second response time [09:21:33] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.042 second response time [09:22:04] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:22:16] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [09:22:25] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:22:33] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.241 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:22:38] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3414 bytes in 1.032 second response time [09:22:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.054 second response time [09:23:09] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:23:31] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 22% [09:23:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:21] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:25] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:24:26] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.140 second response time [09:24:49] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:04] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.056 second response time [09:25:55] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:26:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.86, 1.80, 1.52 [09:26:21] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [09:26:40] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.089 second response time [09:26:58] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.058 second response time [09:26:58] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [09:27:05] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:27:14] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [09:27:27] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:33] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:40] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:44] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3414 bytes in 0.015 second response time [09:28:11] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:28:41] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.024 second response time [09:29:01] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [09:29:12] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:29:17] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:39] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.031 second response time [09:30:06] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.061 second response time [09:30:11] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.456 second response time [09:31:22] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:31:28] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.289 second response time [09:31:51] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.089 second response time [09:32:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.34, 1.89, 1.62 [09:32:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:32:21] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.016 second response time [09:32:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:32:55] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:33:02] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:33:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:33:48] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0008754730225 secs [09:33:55] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:19] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 20% [09:35:27] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:35:35] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:35:50] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.044 second response time [09:35:55] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:36:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.96, 1.97, 1.71 [09:36:32] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [09:36:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:09] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.082 second response time [09:37:23] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13137 MB (34% inode=96%); [09:37:32] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:39] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.017 second response time [09:38:00] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [09:38:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.38, 2.55, 1.95 [09:38:27] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0003741085529 secs [09:38:36] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3414 bytes in 1.042 second response time [09:38:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.064 second response time [09:39:00] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:39:56] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:17] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:43] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:26] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:34] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:35] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.315 second response time [09:41:47] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.011 second response time [09:41:53] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.81, 1.78 [09:42:21] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [09:42:41] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.052 second response time [09:42:56] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:08] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:10] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.306 second response time [09:43:18] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.055 second response time [09:43:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:21] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:26] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3415 bytes in 0.016 second response time [09:43:34] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35731 bytes in 1.042 second response time [09:44:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.53, 1.68 [09:44:36] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:45:03] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 17% [09:45:21] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:45:56] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.09, 5.66, 3.63 [09:47:06] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.043 second response time [09:47:14] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.312 second response time [09:47:56] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001905530691 secs [09:48:00] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.019 second response time [09:48:25] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:48:43] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:49:01] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:16] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [09:49:20] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:49:25] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [09:49:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.97, 1.57, 1.99 [09:50:22] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [09:50:28] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.064 second response time [09:50:39] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.33, 2.41, 2.03 [09:52:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.97, 3.49 [09:52:39] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:42] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:45] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:53:17] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3415 bytes in 0.016 second response time [09:53:47] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.067 second response time [09:53:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.79, 2.10, 2.10 [09:54:00] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.051 second response time [09:54:17] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.059 second response time [09:54:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 3.36, 3.32 [09:56:10] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:13] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:18] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:53] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.063 second response time [09:57:01] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:57:43] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:47] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:48] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.046 second response time [09:57:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.94, 1.60, 1.90 [09:58:07] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 16% [09:58:14] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13135 MB (34% inode=96%); [09:58:22] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:29] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.061 second response time [09:58:49] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:54] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:55] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:57] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:59:11] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [09:59:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:59:47] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:59:49] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.078 second response time [10:00:11] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:39] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:00:44] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.022 second response time [10:00:54] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.285 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:00:58] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 3.102 second response time [10:01:38] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [10:02:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.81, 1.95 [10:02:16] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:38] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.042 second response time [10:02:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.072 second response time [10:03:21] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.020 second response time [10:03:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:03:58] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -3.379583359e-05 secs [10:05:04] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:05:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 2.75, 2.58 [10:06:20] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:35] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:39] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:19] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.022 second response time [10:07:25] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.33, 2.19, 2.40 [10:07:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.59, 1.47, 1.69 [10:08:06] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.058 second response time [10:08:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 1.83, 1.90 [10:08:33] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.240 second response time [10:08:37] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:08:38] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.027 second response time [10:08:48] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.03, 0.06, 0.02 [10:09:08] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:38] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:54] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:10:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.68, 1.84 [10:10:37] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:10:38] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:04] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [10:11:05] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:11:07] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:11:11] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:11:53] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.055 second response time [10:12:10] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.72, 1.82 [10:12:36] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:13:06] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.070 second response time [10:13:45] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:13:52] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:14:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.65, 1.64, 1.78 [10:14:12] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:13] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.027 second response time [10:15:19] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:25] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:59] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 7.418 second response time [10:16:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:16:20] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.289 second response time [10:16:25] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:45] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:45] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:48] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:17:23] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:17:46] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.040 second response time [10:18:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [10:18:44] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 7.305 second response time [10:18:48] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:18:59] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.021 second response time [10:19:15] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.002854973078 secs [10:19:49] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:56] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:21:01] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.062 second response time [10:22:02] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.316 second response time [10:22:32] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.032 second response time [10:22:54] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.101 second response time [10:23:24] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:24:25] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.035 second response time [10:24:35] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [10:25:12] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 3.333 second response time [10:25:35] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:16] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:27:07] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:19] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:41] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:41] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.184 second response time [10:27:52] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:55] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:08] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:12] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:28:41] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [10:29:01] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:22] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.244 second response time [10:29:29] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:29:49] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.048 second response time [10:29:49] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [10:30:10] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.043 second response time [10:30:42] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:59] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:08] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.050 second response time [10:31:59] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.211 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:32:19] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.04, 0.02 [10:32:25] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.051 second response time [10:33:21] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:49] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:34:29] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:34:53] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.028 second response time [10:35:25] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 1.075 second response time [10:35:32] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.033 second response time [10:35:32] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:47] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.022 second response time [10:35:57] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.088 second response time [10:35:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.50, 1.67 [10:36:26] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:49] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:05] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:06] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.065 second response time [10:37:17] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:43] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.013 second response time [10:37:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:56] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:38:22] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:11] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:14] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:17] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35763 bytes in 0.037 second response time [10:39:18] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [10:39:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 1.037 second response time [10:39:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.13, 2.13, 1.79 [10:40:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.027 second response time [10:40:56] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:41:14] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.01, 0.00 [10:41:17] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [10:41:43] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.032 second response time [10:41:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.96, 1.77 [10:42:26] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:38] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.252 second response time [10:43:35] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:43:54] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 1.95, 1.79 [10:44:09] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.053 second response time [10:44:13] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:15] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:45:14] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.099 second response time [10:45:18] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [10:45:44] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:47] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0008349716663 secs [10:45:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.82, 1.76 [10:46:13] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [10:46:20] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 7.103 second response time [10:47:08] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:47:46] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.018 second response time [10:47:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.51, 1.65 [10:48:15] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:48:15] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:48:19] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.021 second response time [10:48:49] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:24] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.108 second response time [10:49:26] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:27] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:13] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.016 second response time [10:50:22] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.044 second response time [10:50:27] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:51:17] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.232 second response time [10:51:22] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [10:51:31] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.172 second response time [10:51:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.80, 1.73 [10:52:31] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.261 second response time [10:52:36] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:52:37] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:48] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:53:02] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.107 second response time [10:53:51] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:53:55] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.102 second response time [10:53:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 2.24, 1.91 [10:54:01] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:54:02] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:35] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:54:45] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [10:55:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.87, 1.72 [10:55:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [10:55:31] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.025 second response time [10:55:49] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [10:56:13] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13129 MB (34% inode=96%); [10:56:58] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.67, 1.66 [10:57:18] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.065 second response time [10:57:32] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:32] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:35] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:58:06] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:58:35] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:58:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:03] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:59:35] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.045 second response time [10:59:42] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:00:04] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:11] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:14] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:15] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:27] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0002161562443 secs [11:00:27] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:38] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.028 second response time [11:00:39] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [11:01:00] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:01:06] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 7.162 second response time [11:01:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.90, 1.88 [11:02:00] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [11:02:11] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.237 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:02:30] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.015 second response time [11:03:04] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -7.247924805e-05 secs [11:03:05] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:04:04] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:46] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [11:05:02] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:05:05] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 24% [11:05:05] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:05:17] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.121 second response time [11:05:27] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.80, 2.10, 1.95 [11:06:14] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [11:06:14] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.090 second response time [11:06:15] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:07:16] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:07:16] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:07:39] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.038 second response time [11:07:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.76, 1.84 [11:08:36] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:08:37] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:21] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:09:25] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:09:26] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:50] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:56] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:07] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 5.099 second response time [11:10:15] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:46] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:37] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 9.667873383e-05 secs [11:11:58] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.041 second response time [11:11:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.18, 2.01, 1.93 [11:12:03] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 7.289 second response time [11:12:17] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:18] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [11:12:44] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:12:45] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 1.231 second response time [11:13:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.035 second response time [11:13:58] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.153 second response time [11:14:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 2.344 second response time [11:14:29] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 7.195 second response time [11:14:32] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:14:43] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.062 second response time [11:14:49] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.243 second response time [11:15:57] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:13] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:22] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.056 second response time [11:16:31] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:59] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 0.036 second response time [11:17:08] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:17:27] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:18:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [11:18:33] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.166 second response time [11:19:02] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:13] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.035 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:19:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:28] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:34] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.420 second response time [11:19:40] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:42] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:57] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:59] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:59] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:20:40] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:48] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:21:06] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:21:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.67, 1.54 [11:21:27] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.069 second response time [11:21:33] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.162 second response time [11:21:49] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:00] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 24% [11:22:13] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:24] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:33] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:06] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:23:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.21, 1.42, 1.46 [11:23:45] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.025 second response time [11:24:00] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:16] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [11:24:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 0.017 second response time [11:24:34] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 2.169 second response time [11:24:37] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [11:24:42] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.017 second response time [11:24:56] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.031 second response time [11:24:57] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures [11:25:40] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.012 second response time [11:26:13] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:32] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.019 second response time [11:27:56] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:58] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:08] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:21] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:56] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:30:02] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:34] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [11:30:43] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:54] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:56] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:03] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.023 second response time [11:31:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.23, 4.55, 3.05 [11:31:42] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:32:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:39] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.62, 2.89 [11:33:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.46, 1.96, 1.66 [11:33:03] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.084 second response time [11:33:03] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:11] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.020 second response time [11:33:15] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:21] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:33:24] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.098 second response time [11:33:43] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 2.093 second response time [11:33:45] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:49] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.051 second response time [11:34:49] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 3.14, 2.80 [11:34:58] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [11:35:27] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.024 second response time [11:36:09] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.533 second response time [11:36:50] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:19] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:19] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.289 second response time [11:37:47] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 0.017 second response time [11:39:25] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 7.249 second response time [11:39:43] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.045 second response time [11:39:44] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.018 second response time [11:39:48] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.070 second response time [11:43:08] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.228 second response time [11:44:03] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.046 second response time [11:44:15] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:44:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:44:20] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.138 second response time [11:44:35] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:44:57] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:14] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:32] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:34] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:50] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:46:31] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [11:47:14] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:47:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:24] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:47:35] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.062 second response time [11:47:48] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [11:47:52] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.057 second response time [11:48:10] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:49:00] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.072 second response time [11:49:16] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 0.017 second response time [11:49:20] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:21] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:49:21] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.330 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:49:26] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:27] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13124 MB (34% inode=96%); [11:49:46] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:15] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:34] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.267 second response time [11:51:09] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:24] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [11:51:26] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:51:27] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.148 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:51:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:36] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:52:27] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.025 second response time [11:52:43] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:53:03] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.062 second response time [11:53:09] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [11:53:29] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:38] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:53:59] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:11] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.65, 1.86, 1.98 [11:54:27] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:50] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:55:30] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:55:31] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:09] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.251 second response time [11:56:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [11:56:27] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:34] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.086 second response time [11:56:50] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.05, 0.05, 0.01 [11:57:13] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 1.025 second response time [11:57:36] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:49] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:58:02] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:58:06] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:19] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.79, 1.99, 1.96 [11:58:34] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.025 second response time [11:59:13] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:59:35] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:59:37] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 0.020 second response time [11:59:41] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [11:59:53] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0005414187908 secs [12:00:02] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.064 second response time [12:00:22] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13118 MB (34% inode=96%); [12:00:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [12:01:26] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:46] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:53] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.021 second response time [12:01:57] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:02:44] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:02:52] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.035 second response time [12:03:47] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35763 bytes in 7.183 second response time [12:04:01] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.256 second response time [12:04:13] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.023 second response time [12:04:13] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:04:57] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.462 second response time [12:05:20] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.060 second response time [12:05:45] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:06:01] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.180 second response time [12:06:16] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [12:06:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.02, 0.04, 0.04 [12:07:15] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:44] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.034 second response time [12:07:56] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:59] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.80, 1.94 [12:09:14] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.022 second response time [12:09:14] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.104 second response time [12:09:16] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:26] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.309 second response time [12:10:02] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:05] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:38] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:11:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.030 second response time [12:11:33] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:45] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.089 second response time [12:11:59] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:12:02] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:12:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.84, 2.04, 1.99 [12:12:20] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:12:37] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [12:13:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:35] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 3.058 second response time [12:13:49] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:14:21] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:14:24] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.022 second response time [12:14:32] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.067 second response time [12:14:37] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:15:05] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.016 second response time [12:15:06] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:38] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 7.137 second response time [12:15:42] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.33, 1.88, 1.95 [12:16:41] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.047 second response time [12:17:02] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.016 second response time [12:18:07] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [12:18:30] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:18:34] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:19:04] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:19:18] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:20:06] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.09, 2.18, 2.03 [12:20:14] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:20:28] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 22% [12:20:51] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.180 second response time [12:21:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.039 second response time [12:21:15] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:16] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.216 second response time [12:21:26] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:56] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [12:22:00] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.13, 0.05, 0.01 [12:22:14] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.055 second response time [12:23:33] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.326 second response time [12:23:37] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.62, 1.95, 1.97 [12:24:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:24] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:53] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:06] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.11, 2.09, 2.02 [12:26:19] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 3.032 second response time [12:26:20] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [12:26:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 7.304 second response time [12:26:39] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:14] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:24] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.014 second response time [12:28:09] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.94, 1.97 [12:28:28] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:36] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 7.289 second response time [12:29:03] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:29:03] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:07] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:07] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [12:29:15] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [12:29:43] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0008441209793 secs [12:29:50] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:15] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:18] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:29] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.044 second response time [12:30:43] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:45] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:08] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:32:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.07, 1.67, 1.83 [12:32:19] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.037 second response time [12:32:21] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13113 MB (34% inode=96%); [12:32:36] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:43] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.015 second response time [12:32:49] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:33:07] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:31] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 46% [12:33:52] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.058 second response time [12:34:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.68, 1.81 [12:34:09] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.066 second response time [12:34:26] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:34:56] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.065 second response time [12:35:15] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:35:20] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:35:25] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:35:42] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.044 second response time [12:35:52] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:36:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.21, 2.25, 2.00 [12:36:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:38] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:37:21] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [12:37:30] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [12:37:34] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 38% [12:37:59] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.026 second response time [12:38:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.14, 1.82, 1.88 [12:38:23] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [12:38:25] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [12:38:40] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 49% [12:39:07] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0002537667751 secs [12:39:20] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 2.086 second response time [12:39:37] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.048 second response time [12:39:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.16, 9.70, 8.71 [12:39:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:40:35] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [12:40:37] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.097 second response time [12:40:59] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:41:02] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.016 second response time [12:42:20] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:42:27] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.07, 0.03, 0.00 [12:42:31] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:42:54] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:43:11] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.079 second response time [12:43:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:43:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.12, 9.25, 8.83 [12:44:01] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:44:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:44] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:51] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:45:38] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.elgeis.com' expires in 15 day(s) (Fri 02 Sep 2022 12:34:13 GMT +0000). [12:45:44] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [12:45:50] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.33, 1.64 [12:46:35] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d3e667f309d3...f800f9892e7d [12:46:45] [url] Comparing d3e667f309d3...f800f9892e7d · miraheze/ssl · GitHub | github.com [12:46:46] [02miraheze/ssl] 07MirahezeSSLBot 03f800f98 - Bot: Update SSL cert for wiki.elgeis.com [12:46:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:23] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.367 second response time [12:47:46] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.055 second response time [12:47:57] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.058 second response time [12:48:05] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:50] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:03] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.031 second response time [12:49:03] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.044 second response time [12:49:21] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:27] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 7.324 second response time [12:49:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.017 second response time [12:49:47] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.astralprojections.org' expires in 15 day(s) (Fri 02 Sep 2022 12:38:19 GMT +0000). [12:49:57] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:49:59] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.016 second response time [12:50:49] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.021 second response time [12:50:52] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:50:54] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.094 second response time [12:51:19] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 20% [12:51:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f800f9892e7d...153339643316 [12:51:28] [url] Comparing f800f9892e7d...153339643316 · miraheze/ssl · GitHub | github.com [12:51:29] [02miraheze/ssl] 07MirahezeSSLBot 031533396 - Bot: Update SSL cert for wiki.astralprojections.org [12:51:31] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 5.051 second response time [12:51:52] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.033 second response time [12:51:55] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.016 second response time [12:52:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.92, 2.76, 2.07 [12:52:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.30, 10.70, 8.90 [12:53:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:47] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:54:06] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:14] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:16] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 1.041 second response time [12:55:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [12:55:50] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.034 second response time [12:56:24] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.138 second response time [12:56:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 4.00, 3.07 [12:56:47] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 4.90, 3.54 [12:57:14] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:25] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:39] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.69, 1.99, 1.94 [12:58:12] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:23] PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thedev.gq' expires in 15 day(s) (Fri 02 Sep 2022 12:36:20 GMT +0000). [12:58:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.54, 11.63, 9.95 [12:58:30] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.89, 4.26, 3.29 [12:58:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:59:17] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:30] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:59:39] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:14] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:16] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:00:28] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:29] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.036 second response time [13:00:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.77, 3.22 [13:00:47] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled] [13:00:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.86, 3.43 [13:00:55] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 1.041 second response time [13:00:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.12, 0.08 [13:01:33] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 7.254 second response time [13:01:36] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.147 second response time [13:01:45] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.068 second response time [13:02:03] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.064 second response time [13:02:26] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.213 second response time [13:02:30] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:32] PROBLEM - inourownwords.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'inourownwords.online' expires in 15 day(s) (Fri 02 Sep 2022 12:36:55 GMT +0000). [13:02:38] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13116 MB (34% inode=96%); [13:02:38] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.027 second response time [13:02:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 4.43, 3.69 [13:03:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/153339643316...a36a74fe9060 [13:03:42] [url] Comparing 153339643316...a36a74fe9060 · miraheze/ssl · GitHub | github.com [13:03:43] [02miraheze/ssl] 07MirahezeSSLBot 03a36a74f - Bot: Update SSL cert for wiki.thedev.gq [13:04:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.03, 1.82, 1.87 [13:04:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.90, 3.59 [13:04:48] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.063 second response time [13:04:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a36a74fe9060...89f1c91ad42d [13:04:58] [url] Comparing a36a74fe9060...89f1c91ad42d · miraheze/ssl · GitHub | github.com [13:04:59] [02miraheze/ssl] 07MirahezeSSLBot 0389f1c91 - Bot: Update SSL cert for inourownwords.online [13:05:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:54] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:09] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:17] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.53, 9.41, 9.64 [13:06:29] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [13:06:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 3.29, 3.40 [13:07:17] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 2.053 second response time [13:07:23] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [13:07:25] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [13:08:14] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:08:17] PROBLEM - wiki.mostly.vet - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mostly.vet' expires in 15 day(s) (Fri 02 Sep 2022 12:38:42 GMT +0000). [13:08:37] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [13:08:48] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:09:00] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:09:44] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:00] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [13:10:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.81, 1.87 [13:10:19] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 7.289 second response time [13:10:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.26, 10.10, 9.86 [13:10:26] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:04] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [13:11:34] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:11:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:38] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:11:45] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:54] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:06] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.65, 2.29, 2.05 [13:12:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.36, 4.80, 3.92 [13:12:13] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.13, 9.76, 9.77 [13:12:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:54] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:07] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.059 second response time [13:13:21] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:13:33] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.111 second response time [13:13:40] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:45] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.095 second response time [13:13:53] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [13:13:54] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:14:17] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 16% [13:14:17] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Tue 15 Nov 2022 11:46:38 GMT +0000. [13:14:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.61, 3.96 [13:14:27] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.073 second response time [13:15:49] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [13:16:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.83, 3.72 [13:16:16] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:48] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.065 second response time [13:16:59] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:17:47] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.017 second response time [13:18:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 3.38, 3.61 [13:18:22] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:18:43] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [13:18:45] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.017 second response time [13:18:59] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [13:19:02] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.020 second response time [13:19:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.66, 2.65, 3.26 [13:19:59] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:03] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:19] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.076 second response time [13:20:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 2.83, 3.37 [13:21:34] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:21:46] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.173 second response time [13:22:11] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.051 second response time [13:22:40] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:18] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:22] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:23:25] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:23:34] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [13:24:39] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.039 second response time [13:25:07] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:25:22] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001195937395 secs [13:25:37] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:25:42] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.082 second response time [13:25:42] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.157 second response time [13:25:45] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:06] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:13] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:00] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.049 second response time [13:27:46] RECOVERY - wiki.thedev.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thedev.gq' will expire on Tue 15 Nov 2022 12:03:35 GMT +0000. [13:27:55] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:28:00] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [13:28:05] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.076 second response time [13:28:25] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.024 second response time [13:28:50] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:29:58] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.054 second response time [13:30:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.53, 10.12, 9.82 [13:30:20] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:26] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:44] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:41] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:49] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:58] RECOVERY - inourownwords.online - LetsEncrypt on sslhost is OK: OK - Certificate 'inourownwords.online' will expire on Tue 15 Nov 2022 12:04:52 GMT +0000. [13:32:00] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:32:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.43, 9.83, 9.74 [13:33:02] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.029 second response time [13:33:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [13:33:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:59] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:34:06] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [13:34:30] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:33] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:34:39] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:18] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:35:43] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:48] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.223 second response time [13:36:13] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:21] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:54] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:55] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.044 second response time [13:37:02] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.163 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:37:14] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.068 second response time [13:37:33] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.077 second response time [13:37:44] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.051 second response time [13:37:54] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.017 second response time [13:37:59] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [13:37:59] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 3.054 second response time [13:38:17] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:23] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:39] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [13:38:47] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 33% [13:38:59] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 5.383 second response time [13:39:54] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:55] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:59] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:40:17] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:40:49] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.022 second response time [13:41:08] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:46] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0001195371151 secs [13:42:08] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.057 second response time [13:43:11] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:43:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.99, 11.15, 9.76 [13:43:14] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:19] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 33% [13:43:27] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:43:45] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:43:54] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.029 second response time [13:44:18] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:51] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.088 second response time [13:44:53] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.021 second response time [13:45:06] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.016 second response time [13:45:28] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [13:45:34] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:45:44] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.056 second response time [13:46:07] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.18, 10.03, 9.61 [13:48:26] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [13:48:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:48:36] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.145 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:49:37] RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Tue 15 Nov 2022 11:51:22 GMT +0000. [13:50:18] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:32] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:47] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:58] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.066 second response time [13:51:14] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.058 second response time [13:51:40] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:52:15] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.013 second response time [13:53:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.64, 10.22, 10.02 [13:53:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.80, 11.25, 10.16 [13:53:19] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:53:37] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.041 second response time [13:54:19] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.017 second response time [13:54:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:24] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:54:26] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 2.068 second response time [13:54:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 9.26, 9.69 [13:55:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.83, 10.56, 10.05 [13:55:14] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:24] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.105 second response time [13:55:47] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.014 second response time [13:55:50] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.035 second response time [13:55:55] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:58] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:56:38] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:45] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:49] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:55] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:00] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:57:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.01, 11.42, 10.41 [13:57:11] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [13:57:47] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:26] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 2.095 second response time [13:58:46] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:59:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 10.87, 10.32 [13:59:16] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:59:17] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.331 second response time [13:59:27] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.018 second response time [13:59:54] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:59:59] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13105 MB (34% inode=96%); [14:00:33] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.93, 12.47, 10.95 [14:01:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [14:01:50] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [14:01:52] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.030 second response time [14:01:52] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:01:57] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:13] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:32] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:34] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 5.084 second response time [14:02:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.57, 10.94, 10.29 [14:03:11] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:03:17] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.039 second response time [14:03:49] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.036 second response time [14:04:00] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.100 second response time [14:04:15] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:23] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:25] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.026 second response time [14:04:27] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.58, 9.83, 9.98 [14:04:54] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 7.376 second response time [14:05:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.49, 10.21, 10.43 [14:05:18] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:20] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.224 second response time [14:05:30] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.060 second response time [14:05:53] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:29] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [14:06:31] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0008031725883 secs [14:06:37] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [14:07:22] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [14:08:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.87, 4.41, 3.41 [14:08:47] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:56] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.024 second response time [14:09:01] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:09:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.56, 3.15 [14:09:47] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:09:51] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:03] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.048 second response time [14:10:06] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.238 second response time [14:10:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:47] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.026 second response time [14:11:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.90, 2.95 [14:11:36] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:21] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:23] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:31] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 27% [14:12:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.80, 3.39 [14:12:38] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.014 second response time [14:12:47] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:36] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:13:52] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:15] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:23] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.07, 9.58, 10.05 [14:15:00] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:15:17] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:23] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:37] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001050829887 secs [14:15:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.71, 10.68, 10.22 [14:16:21] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:16:59] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 7.133 second response time [14:17:11] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [14:17:12] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.028 second response time [14:17:19] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.040 second response time [14:17:21] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 7.181 second response time [14:17:40] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.032 second response time [14:17:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.70, 9.60, 9.88 [14:17:53] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.018 second response time [14:18:12] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:16] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 2.99, 3.19 [14:18:24] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.045 second response time [14:18:42] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:46] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.075 second response time [14:19:22] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:19:43] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.044 second response time [14:19:49] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.079 second response time [14:19:52] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:19:55] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.165 second response time [14:19:56] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:19:58] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:20] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:40] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.156 second response time [14:21:09] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.016 second response time [14:21:19] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 21% [14:21:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:21:50] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.060 second response time [14:21:54] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.034 second response time [14:22:20] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 1.018 second response time [14:22:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.047 second response time [14:22:33] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.038 second response time [14:22:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:23:48] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 5.146 second response time [14:24:05] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:35] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:46] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:11] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.044 second response time [14:26:21] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 5.102 second response time [14:26:35] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [14:26:41] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:28:01] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:28:30] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 38% [14:28:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:28:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.19, 10.30, 9.62 [14:29:05] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:29:27] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.05, 0.01 [14:29:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.72, 10.95, 9.91 [14:29:57] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.040 second response time [14:30:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:30:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.05, 10.11, 9.64 [14:31:21] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:26] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.67, 9.89, 9.64 [14:32:02] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:28] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:35] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:45] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.089 second response time [14:32:55] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:56] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:18] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.01, 0.00 [14:33:25] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:37] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.042 second response time [14:33:41] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.015 second response time [14:34:03] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [14:34:05] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [14:34:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:34:42] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [14:34:54] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [14:35:57] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [14:36:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.82, 2.00 [14:36:14] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:18] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 19% [14:36:21] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.95, 10.34, 9.85 [14:37:26] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:32] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:35] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.046 second response time [14:38:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 2.02, 2.06 [14:38:17] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:38:18] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [14:38:26] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.090 second response time [14:38:29] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.091 second response time [14:38:34] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.036 second response time [14:38:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 9.67, 9.68 [14:39:55] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [14:40:24] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:41] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.053 second response time [14:40:57] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:50] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:18] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.040 second response time [14:42:27] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.348 second response time [14:42:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 10.67, 10.15 [14:42:42] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 5.088 second response time [14:42:59] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.05, 0.03 [14:43:08] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:43:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:43:59] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:03] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.018 second response time [14:44:04] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:44:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.91, 1.99 [14:44:30] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:44:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.81, 9.96, 9.94 [14:45:06] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 26% [14:45:23] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:56] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:16] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13105 MB (34% inode=96%); [14:46:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.039 second response time [14:46:30] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.055 second response time [14:47:44] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:47:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:58] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:48:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.83, 2.02, 1.99 [14:48:26] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.022 second response time [14:48:35] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.497 second response time [14:49:39] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13100 MB (34% inode=96%); [14:49:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:51:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 5.079 second response time [14:51:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:52:12] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:52:33] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:06] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [14:53:06] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.91, 1.98 [14:54:32] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.037 second response time [14:55:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 1.047 second response time [14:56:24] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:57:03] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:57:55] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:00] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.072 second response time [14:58:22] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.036 second response time [14:58:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:59:14] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:59:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.43, 3.02 [14:59:49] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.97, 1.98 [15:00:13] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [15:00:26] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:00:35] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:57] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:01:14] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:01:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.56, 3.10 [15:01:48] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.212 second response time [15:02:11] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:56] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:57] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:20] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:35] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 3.046 second response time [15:03:54] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:56] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:10] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.031 second response time [15:04:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.09, 12.87, 10.53 [15:04:43] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.124 second response time [15:04:48] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.032 second response time [15:04:56] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [15:04:58] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.95, 3.14 [15:05:08] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:09] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.235 second response time [15:05:09] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:21] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.033 second response time [15:05:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.244 second response time [15:05:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.87, 3.34 [15:05:49] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13099 MB (34% inode=96%); [15:05:56] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.037 second response time [15:06:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 3.049 second response time [15:06:10] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [15:06:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 11.41, 10.28 [15:06:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.77, 3.17 [15:07:25] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 20% [15:08:13] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.97, 10.06, 9.91 [15:08:20] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:40] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:09:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.37, 3.25 [15:09:57] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:21] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:10:41] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3416 bytes in 0.016 second response time [15:10:45] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:10:45] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.049 second response time [15:10:50] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [15:10:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 4.21, 3.45 [15:11:02] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:23] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:12:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 10.75, 10.35 [15:12:19] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.03, 0.01, 0.00 [15:13:01] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:13:33] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.242 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:13:42] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.047 second response time [15:13:45] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.85, 9.54, 9.96 [15:14:33] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [15:14:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.42, 3.52, 3.37 [15:14:49] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 2.050 second response time [15:15:06] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.038 second response time [15:15:13] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:15:17] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0003047585487 secs [15:15:25] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.022 second response time [15:15:54] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:16:05] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.296 second response time [15:16:19] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:37] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:16:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 3.17, 3.25 [15:17:03] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:17:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 3.91, 3.52 [15:17:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [15:18:42] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.500 second response time [15:19:05] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [15:19:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 10.40, 9.47 [15:20:04] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:20:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.90, 10.74, 10.23 [15:20:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.98, 3.61 [15:20:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.87, 3.63 [15:21:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 9.33, 9.19 [15:21:45] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:22:09] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.042 second response time [15:22:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.06, 9.73, 9.93 [15:22:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 4.08, 3.68 [15:22:34] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:44] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:22:59] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:23:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 4.15, 3.76 [15:23:39] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.071 second response time [15:23:40] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:23:42] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 1.045 second response time [15:24:11] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:24:50] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.026 second response time [15:24:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.51, 3.56 [15:25:12] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:25:30] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:26:04] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.149 second response time [15:26:25] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:27:38] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 5.095 second response time [15:27:42] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.039 second response time [15:28:21] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:33] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:28:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.06, 3.39 [15:29:02] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3417 bytes in 0.017 second response time [15:29:09] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [15:29:43] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:30:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:08] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.142 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:30:23] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:30:25] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.265 second response time [15:30:50] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [15:30:58] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:08] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:19] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.049 second response time [15:31:47] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.257 second response time [15:31:50] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:06] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 3.058 second response time [15:32:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.06, 3.43, 3.69 [15:32:33] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:49] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:33:00] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.059 second response time [15:33:05] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.087 second response time [15:33:07] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [15:33:33] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.037 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:33:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:33:47] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:34:04] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:09] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:34:31] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13096 MB (34% inode=96%); [15:34:48] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [15:35:56] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:06] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.038 second response time [15:36:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.07, 3.80, 3.71 [15:36:22] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 5.006 second response time [15:36:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.00, 4.35, 3.78 [15:37:47] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [15:37:53] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.018 second response time [15:38:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.40, 3.57 [15:38:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.36, 10.27, 9.60 [15:39:38] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:40] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.42, 3.54 [15:40:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.41, 11.32, 10.03 [15:40:51] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.065 second response time [15:41:15] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:41:25] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.80, 3.75 [15:42:37] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:45] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:11] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.03, 0.01 [15:43:31] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.233 second response time [15:43:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.27, 3.50 [15:44:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 11.06, 10.28 [15:44:32] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:44:41] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:45:51] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:46:01] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:46:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.68, 11.69, 10.61 [15:46:33] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [15:46:40] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [15:47:12] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 10.66, 10.34 [15:48:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 2.70, 3.28 [15:48:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:49:09] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:49:15] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:18] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.029 second response time [15:49:32] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:11] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:50:45] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:50:55] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 19% [15:51:05] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 2.030 second response time [15:51:19] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:51:21] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 7.276 second response time [15:51:37] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.114 second response time [15:51:41] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:53:02] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.071 second response time [15:53:38] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [15:53:42] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:54:06] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.38, 9.61, 10.11 [15:54:49] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:55:37] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.017 second response time [15:55:46] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.086 second response time [15:55:55] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:08] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:10] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:56:51] PROBLEM - mw142 NTP time on mw142 is WARNING: NTP WARNING: Offset 0.1571834981 secs [15:57:02] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.87, 3.64 [15:57:49] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:58] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:58:07] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:58:13] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:58:41] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:59:27] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:59:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.99, 3.73 [15:59:49] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 1.030 second response time [15:59:58] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.045 second response time [16:00:18] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13099 MB (34% inode=96%); [16:00:19] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.047 second response time [16:00:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 7.194 second response time [16:00:51] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:10] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:01:15] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.204 second response time [16:01:16] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.049 second response time [16:01:28] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:01:43] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 24% [16:01:54] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.057 second response time [16:01:54] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:10] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:44] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.071 second response time [16:02:50] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.063 second response time [16:03:08] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [16:03:26] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001094460487 secs [16:03:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 3.04, 3.40 [16:03:53] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:21] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.318 second response time [16:04:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:11] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:32] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:49] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [16:06:09] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [16:06:41] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.016 second response time [16:07:08] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.032 second response time [16:07:10] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [16:07:23] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:07:28] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:08:49] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:09:22] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [16:09:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:09:42] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.133 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:09:43] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.151 second response time [16:10:16] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:10:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:11:06] RECOVERY - mw142 NTP time on mw142 is OK: NTP OK: Offset -0.01887843013 secs [16:11:19] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.082 second response time [16:12:12] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:12:18] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:52] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:40] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:18] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [16:14:56] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:32] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:56] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:16:24] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:16:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.56, 3.22 [16:16:46] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:57] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:57] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:01] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:25] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:25] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:54] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:18:04] PROBLEM - cp20 Puppet on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:18:24] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.167 second response time [16:18:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [16:18:31] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [16:18:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 3.01, 3.05 [16:18:35] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:51] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:19:02] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:19:10] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:19:13] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:19:22] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.021 second response time [16:19:26] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:19:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [16:19:35] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:19:42] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:19:53] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:19:54] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:20:21] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:20:28] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.282 second response time [16:20:41] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:20:46] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:20:49] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [16:20:54] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.044 second response time [16:21:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.23, 10.71, 9.62 [16:21:14] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:21:23] PROBLEM - cp20 NTP time on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 port 5666: Connection refusedconnect to host 2001:41d0:801:2000::4c25 port 5666: Connection refused [16:22:24] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 20% [16:22:27] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.56, 9.32, 9.23 [16:23:41] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:15] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.041 second response time [16:24:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.38, 3.15 [16:25:11] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.114 second response time [16:26:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.24, 3.13 [16:26:34] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 0.016 second response time [16:26:34] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:54] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:27:07] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:34] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:54] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.055 second response time [16:29:15] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:06] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:30:31] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.034 second response time [16:30:38] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.039 second response time [16:31:00] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:12] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:32:04] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13092 MB (34% inode=96%); [16:33:18] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [16:34:28] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.207 second response time [16:35:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 0.015 second response time [16:36:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.94, 3.45 [16:36:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 4.19, 3.51 [16:36:37] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:15] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:38:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.93, 3.51 [16:39:38] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.013 second response time [16:39:43] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:15] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [16:40:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.08, 3.19 [16:41:41] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:42:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.83, 2.94, 3.21 [16:48:44] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:30] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.198 second response time [16:50:41] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [16:50:57] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:51:20] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:53:07] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [16:53:17] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:24] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.093 second response time [16:54:02] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:16] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:55:14] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:56:02] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:16] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [16:56:58] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.09, 9.53, 8.70 [16:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.43, 3.11 [16:59:34] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:00:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 5.10, 3.81 [17:00:39] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.88, 3.36 [17:01:03] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.083 second response time [17:01:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 9.25, 8.84 [17:02:28] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 7.202 second response time [17:02:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.92, 3.44 [17:03:00] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:04] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:06] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.016 second response time [17:03:08] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [17:04:01] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.024 second response time [17:05:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.94, 10.98, 9.64 [17:05:46] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.96, 3.74 [17:07:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.47, 0.11, 0.04 [17:07:17] PROBLEM - cp20 Puppet on cp20 is WARNING: WARNING: Puppet last ran 1 hour ago [17:07:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.78, 10.15, 9.50 [17:07:31] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:32] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:32] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:07:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:01] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.465 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:08:03] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.087 second response time [17:08:15] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13097 MB (34% inode=96%); [17:08:20] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [17:08:23] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.027 second response time [17:08:25] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.088 second response time [17:08:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:08:32] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [17:08:36] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.158 second response time [17:08:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.30, 3.34 [17:08:43] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35731 bytes in 0.040 second response time [17:09:11] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [17:09:15] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [17:09:25] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:32] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001357465982 secs [17:09:34] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.067 second response time [17:09:45] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:09:51] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.062 second response time [17:10:10] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.047 second response time [17:10:14] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.047 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:10:31] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:39] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:11:10] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.001080691814 secs [17:11:14] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.080 second response time [17:11:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 11.21, 10.13 [17:11:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.18, 13.64, 10.31 [17:11:52] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.159 second response time [17:11:57] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [17:12:05] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.016 second response time [17:12:22] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.049 second response time [17:12:30] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.183 second response time [17:12:31] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.079 second response time [17:12:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.64, 3.51 [17:12:54] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 26% [17:13:17] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.063 second response time [17:13:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.35, 11.54, 9.95 [17:13:56] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [17:14:20] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:14:20] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [17:14:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 2.91, 3.29 [17:14:37] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:14:47] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.031 second response time [17:15:14] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:15:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.93, 9.26, 9.59 [17:15:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:15:36] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:15:41] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:16:32] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:16:48] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [17:17:23] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:17:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.38, 9.49, 9.50 [17:17:37] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.035 second response time [17:17:44] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.016 second response time [17:18:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.58, 2.81, 3.24 [17:19:19] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 5.080 second response time [17:19:26] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:47] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:49] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:03] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.100 second response time [17:20:53] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:21:46] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 35% [17:22:05] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:38] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [17:22:50] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.096 second response time [17:23:18] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13095 MB (34% inode=96%); [17:23:35] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:24:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:24:30] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:24:32] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:24:35] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:24:41] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.044 second response time [17:25:11] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:25:22] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:25:35] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.034 second response time [17:26:03] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 46% [17:26:29] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.043 second response time [17:26:31] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001233428717 secs [17:26:37] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:53] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [17:27:15] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [17:27:26] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.088 second response time [17:27:27] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:59] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:35] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.046 second response time [17:29:04] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [17:29:31] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:29:31] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.024 second response time [17:30:34] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:30:44] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:08] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:12] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:30] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:35] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.040 second response time [17:31:38] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:38] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:37] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.070 second response time [17:32:39] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [17:32:40] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.025 second response time [17:32:43] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:25] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.104 second response time [17:33:32] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [17:33:40] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [17:33:50] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:03] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:20] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:35:13] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [17:35:32] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.320 second response time [17:35:37] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [17:35:55] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 10.38, 9.74 [17:36:09] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:36:16] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:36:16] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 2.038 second response time [17:36:19] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.051 second response time [17:36:37] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [17:37:08] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 2.252 second response time [17:37:51] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [17:37:55] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:38:01] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [17:38:08] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:14] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.028 second response time [17:38:24] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [17:39:59] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0004563629627 secs [17:40:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.96, 9.28, 9.49 [17:40:13] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:50] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:51] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:56] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:41:10] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:41:16] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:42:12] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:42:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 10.22, 9.82 [17:42:53] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 3.056 second response time [17:43:15] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.070 second response time [17:43:27] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.045 second response time [17:44:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.55, 9.51, 9.62 [17:44:48] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:44:57] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:45:58] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:46:15] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.028 second response time [17:46:42] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:48:02] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:15] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.055 second response time [17:48:41] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [17:48:44] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:48:51] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:02] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:25] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:35] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [17:49:39] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 46% [17:49:40] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13091 MB (34% inode=96%); [17:50:08] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3412 bytes in 7.154 second response time [17:50:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.26, 10.77, 10.12 [17:50:48] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.083 second response time [17:50:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:42] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:55] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.281 second response time [17:53:03] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.046 second response time [17:53:44] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.245 second response time [17:53:58] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 38% [17:54:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.55, 8.92, 9.55 [17:54:25] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 42% [17:54:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:47] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:54:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.070 second response time [17:54:52] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:55:44] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:55:59] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:24] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [17:56:31] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:51] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:57:18] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:57:20] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.050 second response time [17:57:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:58:09] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.167 second response time [17:58:11] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.079 second response time [17:58:48] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3413 bytes in 3.047 second response time [17:58:52] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.064 second response time [17:59:06] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.014 second response time [17:59:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 3.040 second response time [17:59:39] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 29% [17:59:55] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:00:18] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.017 second response time [18:00:27] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:38] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:38] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:02] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:34] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 5.211 second response time [18:01:57] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:00] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:24] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.041 second response time [18:02:34] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:38] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.039 second response time [18:02:52] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:16] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.057 second response time [18:03:40] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:46] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:03:47] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:54] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.072 second response time [18:03:54] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [18:04:10] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:04:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:04:32] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [18:04:45] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:51] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.030 second response time [18:05:19] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:23] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:05:40] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:42] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.027 second response time [18:05:55] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.046 second response time [18:06:01] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13089 MB (34% inode=96%); [18:06:12] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 2.028 second response time [18:06:14] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.016 second response time [18:06:14] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [18:06:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.53, 2.88 [18:06:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:07:03] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [18:07:23] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:25] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3412 bytes in 0.015 second response time [18:07:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.044 second response time [18:08:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 2.98, 2.75 [18:09:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:09:44] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:09:53] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:10:00] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.088 second response time [18:10:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.40, 1.95 [18:10:09] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:41] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:41] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:11:50] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:11:51] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.189 second response time [18:12:01] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:12:03] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:12:16] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.066 second response time [18:12:17] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:13:21] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 3.062 second response time [18:13:39] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [18:13:52] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.171 second response time [18:14:08] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.319 second response time [18:14:11] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:14] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 3.041 second response time [18:14:14] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:40] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.238 second response time [18:14:45] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:30] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:16:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.196 second response time [18:16:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.84, 1.74, 1.91 [18:16:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.039 second response time [18:16:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.03, 0.06, 0.06 [18:16:20] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.023 second response time [18:16:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:16:40] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 45% [18:16:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:16:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:17:01] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.053 second response time [18:17:03] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.166 second response time [18:17:03] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:57] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:40] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:45] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:46] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.051 second response time [18:19:06] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3413 bytes in 3.049 second response time [18:19:14] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:50] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:59] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 5.093 second response time [18:20:13] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:20:44] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.042 second response time [18:21:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:21:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.034 second response time [18:21:53] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.038 second response time [18:22:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.92, 1.98, 1.97 [18:22:31] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:37] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:22:56] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.142 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:23:15] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:23:38] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.077 second response time [18:23:38] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:24:17] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.037 second response time [18:24:34] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:24:40] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.034 second response time [18:24:47] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 38% [18:25:53] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.114 second response time [18:25:59] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:26:00] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [18:26:01] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:45] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [18:26:58] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:27:30] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:53] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.072 second response time [18:28:06] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.155 second response time [18:28:08] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [18:28:45] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.017 second response time [18:29:28] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.061 second response time [18:29:45] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:30:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.92, 2.01, 1.91 [18:30:53] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:31:38] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:31:43] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [18:31:48] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:33:27] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:33:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:33:30] PROBLEM - ping6 on cp31 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.17 ms [18:33:36] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:33:43] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:48] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:59] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:35:24] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:35:32] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.02, 0.02 [18:35:35] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [18:35:47] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:35:57] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:14] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.146 second response time [18:36:27] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:09] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:37:30] RECOVERY - ping6 on cp31 is OK: PING OK - Packet loss = 0%, RTA = 78.15 ms [18:37:38] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.035 second response time [18:37:55] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0003713071346 secs [18:38:19] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 2.072 second response time [18:38:29] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.035 second response time [18:38:48] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.090 second response time [18:39:07] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:27] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:39:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 9.96, 8.66 [18:39:46] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:46] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:03] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.067 second response time [18:41:22] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.032 second response time [18:41:27] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [18:41:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.73, 9.81, 8.76 [18:41:31] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:37] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:43] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:04] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.049 second response time [18:42:05] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.023 second response time [18:42:40] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.046 second response time [18:43:06] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13078 MB (33% inode=96%); [18:43:38] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.027 second response time [18:44:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 10.65, 9.01 [18:44:51] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:17] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:45:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.48, 11.86, 9.73 [18:45:34] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:45:49] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:45:51] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3413 bytes in 0.016 second response time [18:46:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.99, 11.39, 9.51 [18:46:36] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:43] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.013 second response time [18:47:00] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:47:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 10.79, 9.59 [18:47:40] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:47:45] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.061 second response time [18:48:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:48:56] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:49:18] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:21] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:49:47] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.035 second response time [18:50:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.35, 9.79, 9.34 [18:50:10] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 2.187 second response time [18:50:13] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.02, 0.00 [18:50:28] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:47] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 25% [18:51:01] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [18:51:03] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.333 second response time [18:51:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 1.038 second response time [18:51:16] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:51:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.52, 9.61, 9.41 [18:51:25] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.092 second response time [18:51:33] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.018 second response time [18:51:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:52:08] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 42% [18:52:10] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [18:53:09] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:53:46] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.049 second response time [18:54:12] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 39% [18:54:37] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:55:11] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:55:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [18:57:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.54, 10.61, 9.97 [18:57:25] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:27] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:35] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.034 second response time [18:57:52] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:53] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [18:57:54] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:58:14] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:21] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:13] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:59:17] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:59:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.10, 9.08, 9.49 [18:59:20] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:59:40] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:59:56] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.050 second response time [19:00:00] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [19:01:14] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [19:02:00] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:02:19] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [19:02:59] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.086 second response time [19:03:21] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:03:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.038 second response time [19:03:28] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:03:45] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:03:54] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.122 second response time [19:03:57] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 1.299381256e-05 secs [19:04:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.087 second response time [19:04:39] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.550 second response time [19:05:11] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:05:31] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.094 second response time [19:06:11] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:20] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 3.057 second response time [19:06:37] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:50] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:52] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:09] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:10] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [19:07:10] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:15] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:18] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:22] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.041 second response time [19:08:06] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [19:08:14] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001190602779 secs [19:08:22] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:08:43] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:52] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.030 second response time [19:09:13] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.239 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:09:15] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [19:09:16] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:46] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.178 second response time [19:10:50] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:08] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:12:19] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 25% [19:12:25] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.046 second response time [19:12:42] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:02] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.022 second response time [19:14:49] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3414 bytes in 7.268 second response time [19:15:08] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:03] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.093 second response time [19:18:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:18:04] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:18:04] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:18:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:18:16] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.034 second response time [19:18:48] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [19:18:54] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:19:05] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [19:19:23] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:19:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [19:20:12] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [19:20:24] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:20:32] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:20:35] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:21:18] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.023 second response time [19:21:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:38] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [19:22:12] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.200 second response time [19:22:14] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:37] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 3.053 second response time [19:22:42] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 7.347 second response time [19:22:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:23:49] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.092 second response time [19:24:28] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.258 second response time [19:24:48] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:24:53] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.024 second response time [19:25:07] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [19:25:23] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.039 second response time [19:25:44] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.017 second response time [19:25:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.27, 2.69 [19:25:52] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:25:54] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:09] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:26:59] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13079 MB (33% inode=96%); [19:27:11] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.075 second response time [19:27:26] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:27:43] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:27:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.70, 4.26, 3.13 [19:27:47] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:28:00] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:28:51] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.038 second response time [19:29:04] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:29:11] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:29:37] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:29:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.98, 3.18 [19:29:57] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.022 second response time [19:30:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.62, 10.59, 9.42 [19:30:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:30:51] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [19:30:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.018 second response time [19:31:02] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.065 second response time [19:31:14] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.065 second response time [19:31:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.43, 4.83, 3.58 [19:32:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.23, 10.05, 9.36 [19:32:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.08, 4.72, 3.37 [19:32:34] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [19:32:36] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:33:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.02, 0.00 [19:33:15] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.71, 9.97, 8.84 [19:35:05] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 2.656 second response time [19:35:16] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.023 second response time [19:35:17] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:44] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.167 second response time [19:36:27] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.030 second response time [19:36:52] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:36:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.80, 8.56 [19:37:21] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:37:23] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:38] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.020 second response time [19:38:04] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.074 second response time [19:38:05] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.89, 3.54 [19:38:23] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:39:09] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.072 second response time [19:39:26] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:40:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.68, 3.26, 3.36 [19:41:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:41:27] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.028 second response time [19:41:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.83, 3.73 [19:42:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.67, 1.63, 1.95 [19:42:15] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [19:42:48] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:44:24] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:37] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:47] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.177 second response time [19:45:15] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:45:37] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:01] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:46:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [19:46:54] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.065 second response time [19:47:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.91, 3.40 [19:47:45] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:48:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.23, 1.95, 1.93 [19:48:38] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:59] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:49:01] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [19:49:12] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:49:51] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:16] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [19:50:30] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:38] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.059 second response time [19:50:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:50:57] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.177 second response time [19:51:05] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.043 second response time [19:51:25] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [19:51:40] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.001026928425 secs [19:51:55] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.074 second response time [19:52:10] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:33] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:53:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:54:14] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:54:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.51, 3.37 [19:54:34] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.046 second response time [19:54:38] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 1.038 second response time [19:54:45] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 3.038 second response time [19:55:25] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:56:02] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [19:56:09] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.29, 3.31 [19:56:30] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [19:57:19] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [19:57:29] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:57:34] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:57:42] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 4.357 second response time [19:57:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.17, 3.26 [19:58:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.90, 1.98 [19:58:20] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:58:31] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.016 second response time [19:58:46] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:59:17] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:59:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:59:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.90, 2.76, 3.11 [20:00:09] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:25] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.104 second response time [20:00:26] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [20:00:43] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:01:39] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:01:53] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.026 second response time [20:02:06] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:02:20] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:03] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.018 second response time [20:03:34] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:03:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:03:38] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:41] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:03:58] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 3.054 second response time [20:04:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.98, 1.97 [20:04:17] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [20:04:36] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:05:06] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:05:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:33] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.189 second response time [20:05:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:05:39] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3414 bytes in 1.047 second response time [20:05:43] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13075 MB (33% inode=96%); [20:06:12] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.012 second response time [20:06:33] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.050 second response time [20:06:37] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:04] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.016 second response time [20:07:18] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3418 bytes in 0.015 second response time [20:08:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.84, 1.91 [20:09:25] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.053 second response time [20:09:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:10:00] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:10:25] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:11:27] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.074 second response time [20:12:14] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.047 second response time [20:12:29] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.049 second response time [20:12:48] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:12:57] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:14:25] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:14:39] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [20:14:55] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.018 second response time [20:14:59] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.058 second response time [20:15:05] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:12] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:15:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [20:16:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 6.11, 3.52, 2.48 [20:17:18] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [20:17:30] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:18:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:35] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:24] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [20:22:05] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:23:18] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:23:20] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:23:35] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.208 second response time [20:24:08] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.076 second response time [20:24:28] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:24:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:25:03] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.039 second response time [20:25:18] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.055 second response time [20:25:23] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 7.201 second response time [20:25:28] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.442 second response time [20:25:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 4.22, 3.46 [20:25:34] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:25:45] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.142 second response time [20:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.34, 4.54, 3.62 [20:26:25] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 32% [20:26:42] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.046 second response time [20:26:48] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:27:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.67, 10.64, 9.29 [20:27:37] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.046 second response time [20:28:15] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:28:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.96, 3.52 [20:28:28] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:29:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.59, 10.03, 9.24 [20:30:31] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [20:31:24] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:32:14] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:17] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 4.47, 3.79 [20:32:40] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.030 second response time [20:33:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.84, 3.61 [20:33:23] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.014 second response time [20:33:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:34:18] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [20:34:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.95, 3.68 [20:35:20] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:35:36] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:36:05] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:37:42] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.047 second response time [20:37:45] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.072 second response time [20:37:53] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.077 second response time [20:38:02] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.054 second response time [20:39:00] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:41:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.76, 3.60 [20:42:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 3.12, 3.39 [20:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.35, 4.38, 3.79 [20:45:10] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:45:32] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:46:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.62, 3.52 [20:48:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.46, 3.53 [20:48:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.22, 3.38 [20:49:14] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:44] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.375 second response time [20:50:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:51:12] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:51:44] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:52:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.16, 3.40 [20:52:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:53:06] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:40] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:53:46] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.252 second response time [20:55:08] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.074 second response time [20:56:11] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [20:57:21] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:57:35] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.46, 3.36 [20:59:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.63, 9.80, 8.94 [20:59:39] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [20:59:49] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.00, 3.21 [21:00:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.25, 10.06, 9.22 [21:01:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.01, 8.60, 8.60 [21:01:12] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:02:18] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:02:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:02:43] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.18, 9.50, 9.12 [21:04:02] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:04:40] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:04:49] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [21:05:02] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.059 second response time [21:07:48] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:08:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:08:17] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:04] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.022 second response time [21:09:29] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:45] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.023 second response time [21:10:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [21:10:16] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3415 bytes in 0.017 second response time [21:10:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.71, 4.65, 3.76 [21:11:41] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.567 second response time [21:12:03] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.67, 3.51 [21:12:41] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:01] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:14:15] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 3.080 second response time [21:14:55] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:05] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:15:16] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.054 second response time [21:15:44] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:50] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:16:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 2.97, 3.26 [21:16:51] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:17:00] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [21:17:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 1.77, 2.00 [21:18:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.023 second response time [21:21:04] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.031 second response time [21:21:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.42, 2.47, 2.21 [21:23:19] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [21:23:38] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:24:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:26:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.47, 3.17 [21:28:03] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [21:28:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.28, 3.14 [21:32:56] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:10] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.084 second response time [21:47:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:49:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:51:10] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw141 [21:53:08] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:57:27] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw141 [21:59:23] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [22:01:21] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw121 [22:02:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [22:03:19] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [22:03:35] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw122 mw131 [22:07:43] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [22:14:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:21:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.26, 10.25, 8.29 [22:21:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.53, 4.83, 3.37 [22:22:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.56, 2.89 [22:22:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.74, 10.97, 8.74 [22:23:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.65, 9.84, 8.37 [22:24:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 4.10, 3.18 [22:24:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 10.31, 8.77 [22:27:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.97, 3.52 [22:28:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.54, 3.18 [22:28:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.00, 9.73, 8.90 [22:30:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.30, 3.14 [22:33:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.90, 3.20 [22:41:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 10.02, 8.93 [22:43:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.31, 9.59, 8.89 [22:45:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.77, 1.47, 1.95 [22:46:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 3.94, 3.47 [22:47:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.64, 1.94 [22:48:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.30, 3.29 [23:00:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.44, 3.17 [23:01:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 4.00, 3.35 [23:02:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.90, 1.64, 1.96 [23:02:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 3.18, 3.10 [23:03:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.84, 3.39, 3.19 [23:04:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.94, 2.11, 2.08 [23:06:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.79, 1.97 [23:10:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.69, 1.95, 2.01 [23:12:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.70, 1.91 [23:15:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.17, 9.70, 8.87 [23:16:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.85, 10.24, 9.21 [23:16:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 3.65, 3.19 [23:18:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.91, 9.99, 9.24 [23:19:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.48, 10.05, 9.26 [23:20:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 2.06, 1.99 [23:20:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.79, 3.36 [23:22:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.39, 1.86, 1.93 [23:22:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.21, 3.20 [23:26:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 1.86, 1.90 [23:26:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.03, 4.30, 3.38 [23:26:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.68, 3.82 [23:27:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 10.52, 9.86 [23:28:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.75, 1.87 [23:30:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.66, 2.17, 2.01 [23:32:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.89, 1.93 [23:32:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.89, 3.79 [23:33:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.71, 11.22, 10.30 [23:35:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.30, 10.97, 10.33 [23:36:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 2.13, 2.01 [23:36:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.70, 3.55 [23:37:20] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki=fijss13wiki --overwrite (END - exit=0) [23:37:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 3.09, 3.35 [23:38:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.62, 1.93, 1.96 [23:38:15] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki=fijss13wiki --overwrite=true (END - exit=0) [23:38:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.03, 3.38 [23:43:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.15, 9.56, 9.99 [23:44:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 1.61, 1.77 [23:56:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.02, 1.73, 1.92