[00:00:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.30, 1.89, 1.78 [00:01:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.70, 3.42 [00:03:00] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 3897 MB (10% inode=96%); [00:03:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 3.10, 3.23 [00:03:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.42, 3.23 [00:04:59] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 10572 MB (27% inode=96%); [00:05:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.10, 3.13 [00:06:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.59, 1.97, 1.88 [00:07:38] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 10869 MB (28% inode=96%); [00:20:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.36, 2.82, 2.17 [00:24:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.96, 1.78, 1.90 [00:28:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.37, 1.70 [00:32:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.76, 3.29, 2.11 [00:32:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.76, 1.79 [00:36:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.45, 1.66 [00:38:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.13, 3.92, 2.89 [00:44:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.83, 3.31, 2.98 [00:45:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.98, 1.87, 1.74 [00:47:26] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.84, 2.05, 1.81 [00:48:00] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.50, 3.13 [00:50:00] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.81, 3.36, 3.13 [00:57:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.63, 1.61, 1.89 [00:58:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.71, 3.22, 2.45 [01:06:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.81, 1.99 [01:10:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.83, 2.54, 2.23 [01:22:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 2.99, 2.43 [01:28:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.34, 3.84, 2.94 [01:29:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.37, 10.09, 8.72 [01:30:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.68, 2.00 [01:33:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.73, 3.01 [01:34:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.95, 3.35 [01:34:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 1.84, 1.97 [01:35:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.76, 9.72, 9.08 [01:35:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 3.35, 2.96 [01:38:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 3.10, 3.16 [01:38:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.67, 1.89 [01:44:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 4.26, 3.67 [01:44:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.06, 1.69, 1.98 [01:44:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 2.01, 1.94 [01:46:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.66, 1.82 [01:48:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.06, 3.50, 3.56 [01:48:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.34, 1.96, 2.02 [01:52:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.01, 1.77, 1.95 [01:52:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.64, 1.91, 1.84 [01:56:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.46, 3.48 [01:57:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.75, 3.25 [01:58:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.66, 1.15, 1.61 [01:58:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.87, 1.89 [01:59:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.53, 3.22 [02:00:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.51, 3.51 [02:01:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.13, 3.25, 3.14 [02:04:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.13, 3.36 [02:04:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.98, 2.03, 1.93 [02:06:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.12, 4.67, 2.87 [02:10:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.43, 10.45, 9.08 [02:14:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.49, 10.04, 9.27 [02:32:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.70, 1.96 [02:34:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.12, 1.89, 2.00 [02:36:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.57, 1.87 [02:42:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.59, 1.42, 1.70 [02:48:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.66, 1.71 [02:50:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.61, 1.69 [02:57:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.60, 4.57, 2.99 [02:58:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.17, 6.92, 4.33 [02:58:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 10.31, 8.51 [03:02:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.33, 11.30, 9.25 [03:03:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.95, 12.31, 9.57 [03:04:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.69, 11.08, 9.42 [03:05:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.49, 11.86, 9.80 [03:06:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.80, 9.68, 9.12 [03:07:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.38, 10.03, 9.38 [03:10:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.75, 3.44, 3.98 [03:11:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.45, 3.63 [03:13:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.39, 4.72, 4.06 [03:14:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 4.18, 4.17 [03:17:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.98, 1.74 [03:17:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.47, 3.71 [03:18:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.93, 3.16, 3.77 [03:19:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.79, 1.70 [03:21:08] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 2.08, 1.81 [03:21:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 2.89, 3.39 [03:23:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.87, 1.64, 1.69 [03:24:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.08, 2.60, 3.31 [03:29:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.87, 3.63 [03:37:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.09, 3.14, 3.39 [03:50:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.94, 2.03, 1.72 [03:52:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.68, 1.63 [04:06:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.72, 3.70, 2.73 [04:08:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.58, 2.81, 2.52 [04:12:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.44, 2.06, 1.80 [04:19:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.53, 1.82, 2.00 [04:21:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.44, 2.10, 2.08 [04:22:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.97, 1.76, 1.85 [04:30:41] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.14, 1.43, 1.66 [04:31:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.16, 1.70, 1.92 [04:39:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.73, 1.80 [04:45:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.49, 1.50 [04:49:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.99, 1.95 [04:51:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.55, 2.51, 2.13 [04:53:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.55, 2.15, 1.84 [04:55:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.69, 1.70 [05:00:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:02:17] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw121 mw141 [05:03:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.16, 1.60, 1.89 [05:03:20] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:05:47] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [05:05:57] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw142 [05:06:24] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:10:19] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw131 [05:11:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.22, 1.89, 1.87 [05:13:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.77, 1.82 [05:13:57] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:14:13] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:16:10] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.024 second response time [05:17:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.36, 1.65 [05:18:25] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:20:00] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw121 mw131 [05:20:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.09, 1.71, 1.60 [05:20:35] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [05:21:27] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:23:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.93, 1.79 [05:23:23] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.013 second response time [05:24:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.96, 1.80 [05:25:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.67, 1.71 [05:26:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 2.00, 1.83 [05:27:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.43, 1.61 [05:28:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.11, 1.76, 1.77 [05:29:47] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.57, 1.70 [05:31:25] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:33:41] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.025 second response time [05:34:29] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.91, 1.82 [05:36:29] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.91, 1.53, 1.69 [05:38:43] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:38:52] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:40:14] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [05:40:42] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:40:50] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.145 second response time [05:42:38] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw121 mw131 mw132 [05:43:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [05:45:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.91, 2.56, 1.90 [05:45:46] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.054 second response time [05:49:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.84, 1.74 [05:49:59] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:51:02] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:51:57] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.074 second response time [05:53:07] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:53:14] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 12% [05:53:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.95, 3.04, 2.20 [05:55:05] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:55:26] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:34] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:55:36] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.048 second response time [05:56:04] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:56:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:58:02] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [05:58:05] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.020 second response time [05:59:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.33, 2.56 [06:01:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.82, 2.90, 2.50 [06:02:03] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:02:04] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:02] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [06:03:57] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3341 bytes in 0.015 second response time [06:04:31] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 25% [06:04:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:57] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [06:06:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3330 bytes in 1.038 second response time [06:08:39] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.091 second response time [06:08:43] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:09:07] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:13:23] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.022 second response time [06:17:54] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:50] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:18:53] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:18:56] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:06] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:15] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:20:18] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:20:56] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:21:02] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13248 MB (34% inode=96%); [06:21:02] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3343 bytes in 0.016 second response time [06:21:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.06, 0.01, 0.00 [06:21:09] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.29, 1.90, 1.33 [06:21:24] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [06:21:27] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.043 second response time [06:21:59] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:22:03] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:23:25] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:13] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.302 second response time [06:25:18] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:57] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [06:27:24] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [06:28:01] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:01] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.038 second response time [06:28:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:28:14] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.020 second response time [06:28:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:28:16] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:07] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:17] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:56] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.020 second response time [06:30:08] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.021 second response time [06:30:10] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001220464706 secs [06:31:09] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.68, 1.97, 1.68 [06:31:16] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:17] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.196 second response time [06:33:12] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 0.016 second response time [06:33:17] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:26] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.096 second response time [06:33:32] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 5.074 second response time [06:35:09] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.52, 1.68, 1.63 [06:36:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.86, 3.39, 2.07 [06:36:47] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:36:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:36:58] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:38:26] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:52] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.10, 0.07, 0.07 [06:39:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.86, 2.00 [06:39:42] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:41:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.81, 2.11, 2.07 [06:41:27] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:39] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.056 second response time [06:42:52] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:53] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:43:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.09, 2.63 [06:43:26] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 3.045 second response time [06:44:09] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:44:16] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.089 second response time [06:44:30] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:45:00] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.69, 2.54 [06:46:15] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:18] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 2.052 second response time [06:47:01] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.029 second response time [06:47:27] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:52] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.142 second response time [06:48:20] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.053 second response time [06:49:16] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:49:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.37, 2.84 [06:51:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.31, 2.88 [06:51:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.01, 2.45, 1.79 [06:52:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.59, 3.67, 3.54 [06:52:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.050 second response time [06:53:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.84, 1.96 [06:53:22] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.000867754221 secs [06:53:39] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:54:21] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.45, 2.84, 3.25 [06:54:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:05] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [06:55:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.53, 1.96, 1.75 [06:55:34] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 12% [06:55:58] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [06:57:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.51, 2.22, 1.87 [06:58:49] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:58:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 0.016 second response time [06:59:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.77, 1.75 [06:59:59] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.014 second response time [07:01:05] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.068 second response time [07:01:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.44, 2.18, 1.89 [07:01:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:03:06] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [07:03:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.83, 1.85 [07:04:13] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.75, 1.80 [07:05:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.53, 1.74, 1.81 [07:05:33] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:14] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:06:27] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.014 second response time [07:07:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.79, 1.80 [07:07:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 2.19, 1.99 [07:08:10] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.121 second response time [07:09:13] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.39, 1.71, 1.77 [07:09:14] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.42, 1.86, 1.89 [07:09:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 1.035 second response time [07:10:32] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:47] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:07] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:13] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.99, 1.44, 1.67 [07:11:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.98, 1.93 [07:11:33] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.092 second response time [07:12:00] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:13:08] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:13:20] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 2.071 second response time [07:13:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:14:34] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 0.018 second response time [07:14:39] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:02] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [07:15:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.66, 1.71 [07:15:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.75, 1.87 [07:15:34] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:58] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.021 second response time [07:16:09] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:16:29] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.032 second response time [07:16:34] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:16:55] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:06] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.039 second response time [07:17:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.66, 1.60, 1.68 [07:17:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.99, 1.94 [07:18:04] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:18:33] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005545318127 secs [07:19:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.92, 1.93 [07:19:49] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:19:57] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:19:59] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [07:20:03] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.097 second response time [07:20:42] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:21:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 2.08, 1.98 [07:21:17] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:51] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:22:02] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:22:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:22:18] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:22:50] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:08] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.000433832407 secs [07:23:55] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:24:01] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:24:32] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.45, 0.12, 0.04 [07:24:34] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:24:47] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 1.018 second response time [07:25:19] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:25:22] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:25:39] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:25:59] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:07] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.202 second response time [07:26:10] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.041 second response time [07:27:14] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.020 second response time [07:27:18] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.020 second response time [07:27:38] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [07:28:44] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:28:48] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:58] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.033 second response time [07:29:02] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:00] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [07:30:42] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.017 second response time [07:30:48] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 1.035 second response time [07:30:55] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.067 second response time [07:31:10] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:31:31] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 31% [07:32:49] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.037 second response time [07:33:01] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:10] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.056 second response time [07:33:47] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:02] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:14] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:35:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.82, 1.98 [07:35:17] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:53] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.062 second response time [07:36:11] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:36:25] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:36:59] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 1.034 second response time [07:37:04] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:37:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.74, 2.04, 2.03 [07:37:16] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.255 second response time [07:37:54] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [07:38:13] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:16] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.02, 0.00 [07:38:59] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.070 second response time [07:39:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.71, 1.65 [07:39:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.46, 1.84, 1.96 [07:39:21] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.016 second response time [07:39:31] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [07:40:13] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:19] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:41:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.49, 1.65, 1.63 [07:41:29] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:42:13] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:42:43] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.061 second response time [07:42:47] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [07:42:54] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.020 second response time [07:43:07] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:43:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:19] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:40] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.032 second response time [07:45:06] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13250 MB (34% inode=96%); [07:45:21] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.266 second response time [07:46:07] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [07:46:19] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:46:51] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 0.015 second response time [07:47:26] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:47:35] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.024 second response time [07:48:45] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [07:48:57] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 2.27, 1.89 [07:49:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.23, 2.30, 2.00 [07:49:27] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 1.031 second response time [07:49:28] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:55] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:56] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.034 second response time [07:50:24] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:50:38] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:08] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:58] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.036 second response time [07:52:19] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.272 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:52:28] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0005674660206 secs [07:52:41] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [07:54:13] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:55:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.20, 1.80, 1.84 [07:55:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3343 bytes in 0.016 second response time [07:55:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.068 second response time [07:56:07] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.014 second response time [07:56:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:44] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:56:49] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:57:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.28, 2.19, 1.97 [07:57:05] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.90, 1.99 [07:57:42] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.018 second response time [07:58:49] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:59:00] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 1.80, 1.85 [07:59:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.92, 2.15, 2.06 [07:59:18] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.059 second response time [07:59:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:59:35] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:00:35] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:00:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:01:01] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.143 second response time [08:01:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.82, 2.16, 1.99 [08:01:56] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [08:02:50] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.86, 1.90 [08:03:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:50] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.018 second response time [08:04:04] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.074 second response time [08:04:04] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:05:18] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.086 second response time [08:05:29] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 0.015 second response time [08:05:43] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:05:52] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:22] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:32] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.032 second response time [08:06:47] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 1.79, 1.83 [08:07:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [08:07:38] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:08:06] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.020 second response time [08:08:13] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:40] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:08:48] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.111 second response time [08:09:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.93, 1.47, 1.71 [08:09:07] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.034 second response time [08:09:23] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:54] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:10:10] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.142 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:10:42] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:10:53] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:11] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [08:11:21] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.244 second response time [08:11:44] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [08:11:47] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:12:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 0.018 second response time [08:12:15] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:12:42] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:13:36] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.049 second response time [08:13:46] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.017 second response time [08:14:18] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:14:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:14:55] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 18% [08:15:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.44, 1.46, 1.66 [08:15:14] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.022 second response time [08:15:14] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:15:21] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:15:50] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:26] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.049 second response time [08:16:43] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:45] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:58] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:17:04] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:17:11] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.029 second response time [08:17:23] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.063 second response time [08:17:49] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.019 second response time [08:18:01] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:18:24] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001534491777 secs [08:18:32] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:19:06] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [08:19:21] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:19:29] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 5.143 second response time [08:19:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:20:02] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.072 second response time [08:20:26] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:31] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:38] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:21:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.71, 2.29, 1.93 [08:21:01] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:21:13] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [08:21:47] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [08:21:48] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:21:54] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.03, 0.03 [08:22:23] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [08:22:42] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 7.251 second response time [08:22:48] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:23:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.85, 1.81 [08:23:31] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [08:24:03] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:24:16] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13249 MB (34% inode=96%); [08:24:18] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:24:25] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:24:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.015 second response time [08:24:47] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:25:03] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:25:04] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.025 second response time [08:25:12] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:26:18] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:26:25] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.036 second response time [08:26:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [08:26:59] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [08:27:29] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:27:37] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.487 second response time [08:27:49] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.286 second response time [08:27:51] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:01] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:04] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:09] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:15] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.096 second response time [08:28:45] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:29:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.63, 1.25, 1.57 [08:29:24] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:29:59] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [08:30:16] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 4.462 second response time [08:30:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:30:32] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0004644095898 secs [08:30:42] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.131 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:30:55] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.014 second response time [08:31:17] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.108 second response time [08:31:47] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:58] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.015 second response time [08:32:07] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:14] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 25% [08:32:29] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:54] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.050 second response time [08:32:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:56] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.86, 1.76 [08:33:06] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [08:33:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.71, 1.98 [08:33:15] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.052 second response time [08:33:31] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [08:33:36] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:15] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:39] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:35:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.93, 1.79 [08:35:28] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:03] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.099 second response time [08:36:10] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [08:36:20] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.081 second response time [08:36:29] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.059 second response time [08:36:36] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0004834830761 secs [08:36:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 1.025 second response time [08:37:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.57, 1.67 [08:37:08] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:37:12] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 30% [08:37:31] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [08:39:37] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [08:41:49] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:42:03] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:42:13] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:42:35] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:42:50] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:55] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:06] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:09] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:53] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:53] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:02] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13246 MB (34% inode=96%); [08:44:32] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [08:44:37] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:45] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.025 second response time [08:45:02] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:45:07] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:45:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 6.59, 2.73, 2.13 [08:45:19] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:38] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.012 second response time [08:45:51] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.156 second response time [08:46:02] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.075 second response time [08:46:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:46:43] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.158 second response time [08:47:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.80, 2.32, 1.84 [08:47:02] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:47:11] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [08:47:17] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.082 second response time [08:48:24] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:48:56] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 3.052 second response time [08:49:15] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:49:17] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:49:27] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:35] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [08:50:08] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:50:11] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:50:28] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.067 second response time [08:50:37] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [08:50:41] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.048 second response time [08:51:30] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:31] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:51:58] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.050 second response time [08:52:04] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [08:52:12] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:13] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:33] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:40] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 17% [08:53:37] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:53:48] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:07] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [08:54:24] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.018 second response time [08:54:43] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.034 second response time [08:55:50] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:17] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:50] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.022 second response time [08:57:16] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.057 second response time [08:57:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [08:57:44] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:57:51] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:38] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [08:59:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:53] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:07] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.043 second response time [09:01:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 3.053 second response time [09:01:50] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 7.203 second response time [09:03:05] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:07] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:53] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:03:55] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:58] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0006954669952 secs [09:05:17] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [09:05:34] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:54] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 1.042 second response time [09:05:59] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:08] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 7.205 second response time [09:06:19] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:46] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:03] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:09] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:27] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:32] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:38] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:08:00] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.240 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:08:57] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.021 second response time [09:09:08] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:09] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.044 second response time [09:09:18] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.104 second response time [09:09:22] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:39] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13244 MB (34% inode=96%); [09:09:45] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:57] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:10:03] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.063 second response time [09:10:27] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:10:54] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:10:56] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:11:09] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.219 second response time [09:11:39] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.653 second response time [09:11:50] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.059 second response time [09:11:50] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [09:12:16] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.091 second response time [09:12:31] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 2.040 second response time [09:12:50] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.044 second response time [09:12:50] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.071 second response time [09:12:53] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.034 second response time [09:13:12] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:32] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:34] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 1.033 second response time [09:13:37] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:58] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:15:06] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0006318688393 secs [09:15:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.015 second response time [09:15:58] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 36% [09:17:18] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:38] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [09:17:39] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.484 second response time [09:18:31] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:18:51] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:13] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.071 second response time [09:19:23] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:41] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:04] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:12] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:52] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.016 second response time [09:21:08] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:02] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [09:22:09] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.031 second response time [09:22:37] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:09] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.016 second response time [09:23:23] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.150 second response time [09:23:24] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:23:25] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:24:25] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:30] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:25:01] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:19] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.062 second response time [09:25:20] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [09:25:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:25:56] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [09:26:04] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [09:26:35] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:57] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 1.035 second response time [09:27:05] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [09:27:19] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [09:27:48] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.025 second response time [09:28:05] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:33] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:28:58] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:29:04] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:29:04] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:29:10] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:29:25] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.164 second response time [09:30:05] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.038 second response time [09:30:20] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:20] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:44] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.019 second response time [09:31:07] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13242 MB (34% inode=96%); [09:31:12] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.073 second response time [09:32:10] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.052 second response time [09:32:16] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.395 second response time [09:32:16] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [09:32:18] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001764267683 secs [09:32:21] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.030 second response time [09:33:52] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.072 second response time [09:34:01] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:34:08] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:34:41] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.038 second response time [09:36:21] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:13] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:02] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:38:31] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:38:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:39:02] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.049 second response time [09:39:04] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:10] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 1.023 second response time [09:39:18] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:39:21] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:39:22] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:30] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.057 second response time [09:40:29] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13241 MB (34% inode=96%); [09:40:29] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.031 second response time [09:40:59] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:41:11] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:17] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:19] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.059 second response time [09:41:46] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.054 second response time [09:41:51] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:06] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:40] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:04] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:12] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.063 second response time [09:44:04] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [09:44:07] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.363 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:44:09] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:44:36] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:45:16] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:45:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:45:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:35] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:03] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.022 second response time [09:47:07] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:16] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.050 second response time [09:47:32] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.057 second response time [09:47:34] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:48:01] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.036 second response time [09:48:24] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001674383879 secs [09:48:28] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:48:44] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.022 second response time [09:49:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:49:27] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.049 second response time [09:49:32] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:50:27] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:50:58] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.263 second response time [09:51:18] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:51:22] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [09:52:03] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.027 second response time [09:52:32] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.085 second response time [09:52:42] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:53:17] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:07] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:11] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:55:14] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:27] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.017 second response time [09:55:45] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.053 second response time [09:56:14] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.025 second response time [09:57:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [09:57:39] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:23] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:33] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:59:07] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:00:35] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:00:37] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.045 second response time [10:00:40] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:01:40] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:04] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.042 second response time [10:02:27] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:35] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.026 second response time [10:03:10] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:03:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:03:26] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13839 MB (35% inode=97%); [10:03:39] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13240 MB (34% inode=96%); [10:03:46] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.019 second response time [10:03:54] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:04:04] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.032 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:05:36] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.169 second response time [10:05:39] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.036 second response time [10:05:46] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [10:06:12] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 3.037 second response time [10:06:39] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:03] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:07] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:08:02] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:08:36] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.141 second response time [10:08:46] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 13 minutes ago with 0 failures [10:09:02] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:09:08] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.036 second response time [10:09:15] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 26% [10:09:22] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:09] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:16] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:24] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:10:34] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:10:51] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.048 second response time [10:10:57] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:10:59] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.040 second response time [10:11:19] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.117 second response time [10:11:29] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:05] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.014 second response time [10:12:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.016 second response time [10:12:30] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:33] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:43] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:01] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:13:19] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:14:09] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:32] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.028 second response time [10:14:48] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:14:51] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:10] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:20] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:21] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 7.218 second response time [10:15:41] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:44] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0006353259087 secs [10:15:49] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.015 second response time [10:16:48] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.271 second response time [10:17:31] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:18:08] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.243 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:18:25] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13238 MB (34% inode=96%); [10:19:01] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:19:25] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:19:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:19:41] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:19:50] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.077 second response time [10:19:54] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.013 second response time [10:19:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.02, 0.02, 0.01 [10:19:57] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:20:02] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:20:30] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.033 second response time [10:20:36] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [10:20:37] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:21:14] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [10:21:22] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.08, 0.04, 0.01 [10:21:28] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.082 second response time [10:22:23] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:26] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.016 second response time [10:22:30] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 30% [10:22:56] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.051 second response time [10:23:25] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.013 second response time [10:23:30] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.043 second response time [10:23:52] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:24:23] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.051 second response time [10:24:33] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:39] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.029 second response time [10:25:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:25:30] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.030 second response time [10:25:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:25:57] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:58] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.015 second response time [10:26:14] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.280 second response time [10:27:05] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:07] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:21] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:53] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.042 second response time [10:29:54] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:04] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.067 second response time [10:30:18] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:30:26] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:30:50] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:55] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.079 second response time [10:30:56] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:09] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:18] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:37] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:04] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:20] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:40] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 3.054 second response time [10:32:42] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:46] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:26] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:33:40] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:54] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:34:05] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:34:10] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:34:16] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100151 bytes in 1.186 second response time [10:34:19] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.047 second response time [10:34:21] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.023 second response time [10:34:37] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.044 second response time [10:34:52] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.017 second response time [10:35:11] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.055 second response time [10:35:11] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:26] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.057 second response time [10:35:36] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [10:35:49] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.194 second response time [10:36:04] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [10:36:09] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:28] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 18% [10:36:30] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:09] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.159 second response time [10:38:24] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.055 second response time [10:39:24] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [10:39:57] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:09] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:40:39] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.014 second response time [10:40:51] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:08] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:41:52] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:12] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 1.057 second response time [10:42:13] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:25] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:27] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:14] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [10:43:25] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:43:54] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.021 second response time [10:43:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:25] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13238 MB (34% inode=96%); [10:44:35] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:44:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:45:22] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:38] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:45:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.02, 0.01, 0.00 [10:45:59] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.050 second response time [10:46:12] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:47:22] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.020 second response time [10:47:43] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:08] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.064 second response time [10:48:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:31] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [10:48:35] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.017 second response time [10:48:45] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:49] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:06] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:49:07] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.085 second response time [10:49:09] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.040 second response time [10:49:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:49:26] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:57] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.014 second response time [10:50:18] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:24] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:29] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.023 second response time [10:50:32] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:50:34] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [10:51:19] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.013 second response time [10:51:24] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:52:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 1.050 second response time [10:52:13] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [10:52:26] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [10:52:57] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.033 second response time [10:52:59] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.050 second response time [10:53:20] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:54:17] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.033 second response time [10:54:48] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:55:59] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:56:12] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.025 second response time [10:56:44] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:56:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [10:56:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:01] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:45] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:58:26] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:53] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.018 second response time [10:58:59] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.028 second response time [10:59:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:59:18] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.08, 9.27, 7.65 [11:00:12] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13235 MB (34% inode=96%); [11:00:19] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.068 second response time [11:00:20] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.024 second response time [11:00:34] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:55] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [11:00:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.26, 0.24, 0.10 [11:01:07] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:01:11] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:01:15] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:02:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.31, 9.12, 7.79 [11:02:43] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:05] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:19] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0004924833775 secs [11:04:24] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.094 second response time [11:04:55] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [11:05:06] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:05:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.014 second response time [11:05:54] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:06:23] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:07:08] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [11:08:08] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:08:11] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:08:21] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:08:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:50] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:03] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:09:29] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.013 second response time [11:10:40] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [11:10:50] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 30% [11:11:01] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [11:11:02] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13234 MB (34% inode=96%); [11:11:07] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:21] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.021 second response time [11:11:25] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:33] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:47] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:37] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.025 second response time [11:13:06] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.059 second response time [11:13:35] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.031 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:14:01] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.018 second response time [11:14:02] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:33] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.047 second response time [11:15:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:03] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.023 second response time [11:16:15] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [11:16:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:23] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.056 second response time [11:16:37] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.021 second response time [11:18:20] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:18:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.02, 0.00 [11:18:59] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:16] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:33] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [11:19:34] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:20:14] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.052 second response time [11:20:18] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:20:19] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.185 second response time [11:20:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:25] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.026 second response time [11:21:31] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.028 second response time [11:22:12] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:14] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:46] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:10] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [11:24:19] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:38] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:43] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:47] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.051 second response time [11:25:21] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:25:28] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:25:41] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:25:47] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 27% [11:26:07] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:26:58] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.019 second response time [11:27:02] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.047 second response time [11:27:16] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:21] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.019 second response time [11:27:23] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:27:32] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:27:36] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.020 second response time [11:28:15] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:31] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:55] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:35] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.039 second response time [11:29:46] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.035 second response time [11:29:49] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [11:30:01] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:30:26] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:30:28] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [11:30:39] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.059 second response time [11:30:48] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.070 second response time [11:31:12] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.073 second response time [11:32:24] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.137 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:32:51] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:10] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:33:10] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:34:18] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.015 second response time [11:34:50] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.041 second response time [11:34:57] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.192 second response time [11:35:23] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:19] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:31] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:36:34] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:37:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:38:20] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.015 second response time [11:38:23] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.047 second response time [11:38:35] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:38:58] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:06] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:22] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:38] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:55] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [11:40:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:40:26] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.041 second response time [11:40:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:40:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.26, 9.96, 9.06 [11:40:42] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:55] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.216 second response time [11:40:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 1.044 second response time [11:41:13] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 3.078 second response time [11:41:33] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.053 second response time [11:41:33] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:21] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.058 second response time [11:42:57] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [11:43:04] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:43:43] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13233 MB (34% inode=96%); [11:44:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 9.09, 8.95 [11:44:48] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:03] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.018 second response time [11:45:20] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:46:49] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [11:47:08] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.021 second response time [11:47:16] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:47:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:47:46] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:48:30] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:49:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:09] PROBLEM - grk.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address grk.archiopedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:49:18] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.018 second response time [11:49:18] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:14] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.023 second response time [11:50:57] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [11:51:05] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.014 second response time [11:51:14] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.085 second response time [11:51:14] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:51:24] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:53] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:52:41] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.186 second response time [11:53:01] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:53:10] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 3.039 second response time [11:53:36] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [11:53:40] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.052 second response time [11:53:48] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:07] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:19] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 23% [11:56:04] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:25] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.039 second response time [11:56:38] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.017 second response time [11:57:01] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:58:01] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [11:58:03] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.062 second response time [11:58:14] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.064 second response time [11:58:56] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.015 second response time [11:58:59] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:59:01] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.021 second response time [11:59:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [11:59:54] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:00:07] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:30] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:39] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:47] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:55] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13231 MB (34% inode=96%); [12:01:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:02] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.047 second response time [12:03:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:04:00] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:06] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:04:47] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.063 second response time [12:05:49] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 7.354 second response time [12:06:32] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:06:46] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.017 second response time [12:08:20] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [12:08:30] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.253 second response time [12:08:53] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.049 second response time [12:08:55] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0003405511379 secs [12:09:15] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:09:20] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 23% [12:11:10] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:17] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:05] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:07] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:12] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:13:35] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:14:30] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:43] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.055 second response time [12:15:11] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.015 second response time [12:16:28] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.435 second response time [12:16:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:17:20] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:26] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:18:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.034 second response time [12:18:18] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:18:38] RECOVERY - grk.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.archiopedia.org' will expire on Sun 02 Oct 2022 05:34:49 GMT +0000. [12:18:49] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:19:26] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:26] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.089 second response time [12:20:18] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.074 second response time [12:20:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:20:31] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:23] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [12:21:36] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:43] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:46] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:14] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:15] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:28] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.155 second response time [12:22:37] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.043 second response time [12:23:18] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.021 second response time [12:23:32] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:23:37] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:23:48] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.029 second response time [12:24:48] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:14] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:34] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:44] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.053 second response time [12:26:27] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.120 second response time [12:26:28] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [12:26:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:26:50] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:13] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.547 second response time [12:27:35] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 7.093 second response time [12:27:38] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [12:27:40] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [12:28:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:00] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:05] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:16] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:16] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:29:05] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.064 second response time [12:29:19] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 17% [12:29:32] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:16] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.01, 0.03, 0.00 [12:30:25] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:42] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:31:43] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:46] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [12:31:48] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:31:58] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:19] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:33:08] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 7.141 second response time [12:33:08] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [12:33:20] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:33:29] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13228 MB (34% inode=96%); [12:33:43] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.027 second response time [12:34:00] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.015 second response time [12:34:00] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:23] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.060 second response time [12:34:26] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.040 second response time [12:34:49] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:35:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [12:35:36] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [12:36:05] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.016 second response time [12:36:18] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:36:19] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:24] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:31] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:04] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:06] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.069 second response time [12:38:10] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:13] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [12:38:15] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.020 second response time [12:38:27] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001025527716 secs [12:38:28] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.057 second response time [12:38:33] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:40:15] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:40:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.573 second response time [12:41:07] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [12:41:25] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:26] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:43:33] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 3.043 second response time [12:44:57] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:45:07] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:45:11] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:45:27] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 0.020 second response time [12:45:48] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13823 MB (35% inode=97%); [12:46:02] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [12:46:14] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.062 second response time [12:46:17] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 14% [12:46:25] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.050 second response time [12:46:36] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:09] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [12:47:27] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.042 second response time [12:47:40] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:49:06] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [12:49:47] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 29% [12:49:55] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:22] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:51:51] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.086 second response time [12:52:10] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.044 second response time [12:52:32] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:40] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.286 second response time [12:52:50] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:52:59] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.047 second response time [12:53:08] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:53:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 3.27, 2.24 [12:53:22] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:53:34] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:53:45] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:15] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:00] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.261 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:55:17] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.06, 0.05, 0.05 [12:55:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:53] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.77, 12.17, 9.41 [12:56:10] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.102 second response time [12:56:18] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.89, 2.99 [12:56:28] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:29] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:12] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:25] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:55] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:58:09] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:10] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:15] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.061 second response time [12:58:17] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 30% [12:58:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.55, 5.69, 3.79 [12:58:21] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.49, 13.06, 9.50 [12:59:05] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:59:08] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:59:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 1.025 second response time [12:59:58] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.030 second response time [13:00:04] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:16] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:17] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.271 second response time [13:00:20] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.019 second response time [13:00:37] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 1.029 second response time [13:00:50] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.051 second response time [13:00:56] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:19] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001454800367 secs [13:01:25] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.159 second response time [13:01:31] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:37] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.025 second response time [13:01:52] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [13:02:00] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.023 second response time [13:02:24] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:26] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [13:02:26] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 3.076 second response time [13:02:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.33, 11.23, 9.62 [13:02:50] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:59] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.075 second response time [13:03:04] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.850 second response time [13:03:29] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.126 second response time [13:04:27] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.390 second response time [13:05:20] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.04, 0.03, 0.01 [13:05:24] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:18] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [13:06:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.73, 11.31, 9.97 [13:06:33] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [13:06:59] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.017 second response time [13:07:47] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:08:30] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.021 second response time [13:08:35] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:08:55] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:09:31] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 2.342 second response time [13:09:59] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:10:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:18] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.064 second response time [13:10:55] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [13:11:15] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [13:11:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.77, 3.73 [13:11:33] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:11:38] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 11.85, 11.81 [13:12:02] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 1.029 second response time [13:12:14] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [13:12:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:37] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:13:33] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.016 second response time [13:13:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.07, 11.98, 11.09 [13:14:14] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:14:15] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:34] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [13:14:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.022 second response time [13:15:38] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:15:46] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.114 second response time [13:15:54] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:29] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:16:38] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:16:47] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:14] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:17:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.45, 3.56 [13:17:39] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.071 second response time [13:17:39] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:17:40] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:18:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [13:18:43] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:09] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:19:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.92, 3.35 [13:19:28] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:19:32] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:08] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 26% [13:20:35] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:21:08] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.038 second response time [13:21:28] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13225 MB (34% inode=96%); [13:21:35] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [13:21:42] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.060 second response time [13:21:51] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.054 second response time [13:21:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.43, 3.89 [13:22:46] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:23:05] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.79, 8.32, 9.81 [13:24:07] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [13:24:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:37] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:59] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:25:05] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.185 second response time [13:25:12] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:34] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.016 second response time [13:25:40] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.19, 9.32, 10.20 [13:26:05] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:26:39] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 1.025 second response time [13:26:42] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:45] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.183 second response time [13:26:47] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [13:26:47] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.075 second response time [13:27:18] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.041 second response time [13:27:38] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:27:39] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:24] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:29:26] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [13:29:51] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 3.065 second response time [13:30:50] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:55] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:49] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:32:02] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0008668601513 secs [13:32:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 2.98, 3.40 [13:32:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.041 second response time [13:32:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 10.44, 10.38 [13:32:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:01] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:33:06] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.141 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:33:08] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 13% [13:33:26] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.247 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:34:54] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:34:55] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 1.018 second response time [13:34:58] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.039 second response time [13:36:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.23, 3.43 [13:36:19] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.99, 9.50, 10.05 [13:36:55] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.064 second response time [13:37:03] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:00] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.685 second response time [13:39:19] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.011 second response time [13:39:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.65, 3.26 [13:40:17] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.063 second response time [13:40:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.05, 3.33 [13:40:20] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:40:22] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:01] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:35] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.57, 3.27 [13:41:51] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.068 second response time [13:42:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:34] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:42:40] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.54, 10.69, 10.36 [13:43:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.25, 3.18 [13:43:54] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:27] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 7.290 second response time [13:44:37] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13224 MB (34% inode=96%); [13:44:43] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [13:45:15] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.020 second response time [13:45:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.29, 9.64, 10.01 [13:45:53] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.072 second response time [13:46:15] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.057 second response time [13:46:35] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:46:40] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:46:59] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:39] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.088 second response time [13:47:44] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.83, 12.23, 10.34 [13:48:51] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:29] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:49:48] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.023 second response time [13:49:51] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:22] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:42] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:50] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:51] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:59] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 3.058 second response time [13:51:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.18, 10.68, 10.27 [13:51:13] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:47] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.026 second response time [13:51:49] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.338 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:51:56] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [13:52:34] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 31% [13:52:46] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [13:52:50] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.028 second response time [13:52:55] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.202 second response time [13:52:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 11.15, 10.53 [13:53:31] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:54:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 11.41, 10.67 [13:54:28] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:54:33] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:54:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.04, 9.81, 10.13 [13:54:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.018 second response time [13:55:06] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:40] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 3.549456596e-05 secs [13:56:31] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.102 second response time [13:56:43] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.038 second response time [13:57:03] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:57:11] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:18] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [13:58:11] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:58:42] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:59:08] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.053 second response time [13:59:15] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:00:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.08, 9.72, 10.12 [14:01:12] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005376636982 secs [14:01:55] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.175 second response time [14:03:26] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:31] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:03:38] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.131 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:04:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.28, 11.63, 10.78 [14:04:16] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:19] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.83, 10.93, 10.30 [14:04:36] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:03] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.038 second response time [14:05:11] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.033 second response time [14:05:25] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.041 second response time [14:05:38] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:05:45] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [14:06:12] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13221 MB (34% inode=96%); [14:06:15] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:06:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:06:39] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 7.084 second response time [14:07:12] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:19] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:11] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [14:09:30] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:09:40] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [14:09:59] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:11:20] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.022 second response time [14:11:43] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:27] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:37] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:13:18] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.196 second response time [14:13:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [14:13:37] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:56] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 11.83, 11.39 [14:14:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:14:38] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.046 second response time [14:14:56] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:15:05] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:15:23] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.098 second response time [14:15:42] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 1.036 second response time [14:16:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 11.64, 11.62 [14:16:05] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.293 second response time [14:16:57] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [14:16:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:09] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13221 MB (34% inode=96%); [14:17:21] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [14:17:29] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [14:17:42] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:17:48] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.20, 11.68, 11.62 [14:18:41] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:53] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:55] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.104 second response time [14:19:18] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:19:34] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:58] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.68, 11.26, 11.19 [14:20:06] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:12] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:20:34] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:36] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:20:46] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:21:00] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 1.038 second response time [14:21:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 11.15, 11.19 [14:22:03] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.037 second response time [14:22:19] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:29] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.033 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:22:33] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [14:22:48] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:22:54] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [14:23:09] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:17] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [14:23:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [14:23:38] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 7.094 second response time [14:24:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 11.33, 11.54 [14:24:03] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 7.231 second response time [14:24:18] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.030 second response time [14:24:50] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.036 second response time [14:25:04] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [14:25:23] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:32] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.016 second response time [14:26:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:41] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [14:27:12] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:27:14] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 8.64, 10.05 [14:28:02] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.027 second response time [14:28:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 11.14, 11.36 [14:28:37] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:29:09] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 3.089 second response time [14:29:19] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:29:21] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:29:22] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.080 second response time [14:29:26] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.031 second response time [14:30:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.18, 10.94, 11.24 [14:30:11] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:07] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:31] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.016 second response time [14:32:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.07, 11.68, 10.96 [14:32:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.57, 11.53, 11.39 [14:32:13] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled] [14:32:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:01] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [14:33:03] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:23] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:25] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:29] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:23] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:35:01] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.018 second response time [14:35:24] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:35:30] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.290 second response time [14:37:11] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.069 second response time [14:37:14] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [14:37:25] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.034 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:37:27] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:37:37] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.022 second response time [14:38:14] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.019 second response time [14:38:55] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.021 second response time [14:39:27] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.056 second response time [14:39:29] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.015 second response time [14:39:47] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:59] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:03] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:17] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 19% [14:40:49] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:51] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:42:41] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:42:44] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.125 second response time [14:43:26] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [14:43:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.018 second response time [14:43:51] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.37, 11.54, 11.92 [14:44:24] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:44:47] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:11] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.049 second response time [14:45:36] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:45:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:21] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:25] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:35] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.239 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:46:45] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.029 second response time [14:47:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:48:03] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:48:56] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [14:49:05] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:35] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.036 second response time [14:49:36] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:51] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3345 bytes in 7.209 second response time [14:50:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.39, 9.66, 11.76 [14:50:10] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:50:46] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:51:01] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:41] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.012 second response time [14:52:04] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.100 second response time [14:52:22] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [14:52:32] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:15] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [14:53:40] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.138 second response time [14:53:45] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.013 second response time [14:54:15] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 39% [14:54:19] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:54:26] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.015 second response time [14:54:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:54:52] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:55:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:55:38] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:55:49] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:00] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.063 second response time [14:56:36] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.016 second response time [14:56:37] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.039 second response time [14:56:50] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0004659593105 secs [14:56:59] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 24% [14:57:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:48] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.161 second response time [14:57:50] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:09] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:58:17] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.088 second response time [14:58:19] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:32] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:58:55] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:59:50] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.04, 0.09, 0.04 [15:00:08] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:17] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.477 second response time [15:00:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:37] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.034 second response time [15:01:40] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:02:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.65, 8.53, 9.86 [15:02:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.43, 8.52, 10.09 [15:02:06] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:02:38] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.058 second response time [15:03:48] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:29] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:53] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:04] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:05:22] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:31] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [15:06:34] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:42] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.052 second response time [15:06:58] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:07:07] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:33] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:09:25] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.050 second response time [15:10:46] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:11:04] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.038 second response time [15:11:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:12:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.89, 11.80, 10.70 [15:12:07] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.081 second response time [15:12:29] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:12:59] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [15:13:53] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.46, 2.67 [15:14:29] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.023 second response time [15:15:00] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:15:01] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 3.077 second response time [15:15:56] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.193 second response time [15:16:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.94, 11.27, 10.76 [15:16:11] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.031 second response time [15:16:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.15, 2.64 [15:16:25] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:17:05] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13811 MB (35% inode=97%); [15:17:06] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.270 second response time [15:17:29] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:17:43] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:19:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.01, 0.01, 0.00 [15:19:20] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:20:01] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:20:07] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:42] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:20:52] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:20:59] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:25] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:21:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:21:41] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:43] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:47] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [15:22:22] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.023 second response time [15:22:48] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:53] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [15:23:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:23:21] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:33] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.053 second response time [15:23:37] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:23:44] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.095 second response time [15:23:58] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:03] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:24:18] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.096 second response time [15:25:15] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.135 second response time [15:25:36] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.036 second response time [15:25:54] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:26:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.080 second response time [15:26:17] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.140 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:26:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:27:25] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 3.033 second response time [15:27:35] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.036 second response time [15:27:58] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.016 second response time [15:28:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.40, 9.43, 10.11 [15:29:21] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:30:04] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [15:30:34] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:10] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:26] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:44] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:48] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:55] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:05] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [15:33:37] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.062 second response time [15:33:45] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:12] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.054 second response time [15:34:51] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [15:35:45] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:49] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 9.82, 9.30 [15:36:46] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.092 second response time [15:36:48] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:52] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.016 second response time [15:37:46] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.016 second response time [15:37:51] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.342 second response time [15:38:04] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.089 second response time [15:38:05] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:38:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.60, 10.96, 9.76 [15:38:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.87, 4.38, 3.49 [15:38:46] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:45] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.56, 11.50, 10.12 [15:40:18] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.058 second response time [15:40:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.77, 3.37 [15:40:24] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:47] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [15:40:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:24] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:41:26] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:41:41] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.071 second response time [15:41:50] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:02] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:26] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.245 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:42:35] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:47] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:47] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:54] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:43:58] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:44:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.59, 9.44, 9.64 [15:44:09] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.18, 4.70, 3.76 [15:44:18] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.103 second response time [15:44:34] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.238 second response time [15:45:38] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [15:45:57] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:45:57] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.061 second response time [15:46:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.90, 3.59 [15:47:11] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.088 second response time [15:47:50] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 22% [15:47:51] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:48:00] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [15:48:49] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.028 second response time [15:48:54] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.069 second response time [15:48:55] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.044 second response time [15:49:03] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:49:09] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.054 second response time [15:49:15] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:48] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [15:50:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.89, 2.91, 3.25 [15:50:30] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:51:11] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.021 second response time [15:51:55] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:52:00] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [15:52:06] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:13] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.015 second response time [15:52:32] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [15:52:44] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [15:53:25] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:53:49] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [15:54:34] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.483 second response time [15:54:57] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 6.610155106e-05 secs [15:55:04] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:55:08] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:55:38] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:57:02] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:08] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.017 second response time [15:57:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.022 second response time [15:57:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:35] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:57:41] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:58:17] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:58:39] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:47] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:53] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:58:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:59:05] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.056 second response time [15:59:30] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:59:40] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:59:54] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:00:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:00:37] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:53] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.051 second response time [16:01:17] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.037 second response time [16:01:27] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [16:01:40] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [16:02:12] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:26] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:41] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.016 second response time [16:02:49] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13212 MB (34% inode=96%); [16:03:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.86, 1.78, 1.97 [16:03:32] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:07] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.024 second response time [16:04:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:32] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:35] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:38] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 3.037 second response time [16:04:39] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.027 second response time [16:05:00] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:02] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:05] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:37] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:45] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:18] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:38] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:45] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [16:06:46] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.016 second response time [16:07:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 1.82, 1.93 [16:07:07] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.021 second response time [16:07:31] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.077 second response time [16:08:45] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.197 second response time [16:09:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.52, 1.77, 1.90 [16:09:01] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:20] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:10:14] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [16:10:28] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [16:10:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.11, 2.79 [16:10:47] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:11:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.72, 2.02, 1.97 [16:11:06] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [16:11:30] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:11:38] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.253 second response time [16:12:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.76, 2.56, 2.62 [16:13:03] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.016 second response time [16:13:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [16:13:18] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.020 second response time [16:13:32] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:13:36] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.065 second response time [16:13:37] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.474 second response time [16:13:56] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:10] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:14:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [16:15:50] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005253553391 secs [16:16:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.40, 10.30, 8.40 [16:16:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 4.02, 3.18 [16:16:21] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13212 MB (34% inode=96%); [16:16:55] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:08] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:26] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:40] [02ssl] 07AgentIsai opened pull request 03#595: Unset redirect for polandballwiki.com - 13https://github.com/miraheze/ssl/pull/595 [16:17:40] [url] Page not found · GitHub · GitHub | github.com [16:17:43] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.33, 10.71, 8.80 [16:18:26] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:51] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 0.015 second response time [16:18:54] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.055 second response time [16:19:10] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.930 second response time [16:19:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.308 second response time [16:19:29] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 7.309 second response time [16:19:32] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.036 second response time [16:19:41] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:19:50] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:20:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 22.96, 14.38, 10.31 [16:20:10] [discord] @CosmicAlpha around perhaps? ^ [16:20:28] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:21:25] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:29] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.045 second response time [16:21:52] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [16:21:56] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:22:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.69, 11.72, 9.96 [16:22:45] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 3.318 second response time [16:23:16] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:23:53] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.70, 11.83, 10.20 [16:24:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.88, 3.61 [16:24:18] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:40] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 40% [16:24:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:45] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [16:24:49] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:25:23] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.287 second response time [16:25:45] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:25:49] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 1.031 second response time [16:26:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.11, 11.31, 10.21 [16:26:06] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:15] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 4.27, 3.80 [16:26:40] [02ssl] 07AgentIsai commented on pull request 03#595: Unset redirect for polandballwiki.com - 13https://github.com/miraheze/ssl/pull/595#issuecomment-1221578116 [16:26:40] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 39% [16:26:40] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:41] [url] Unset redirect for polandballwiki.com by AgentIsai · Pull Request #595 · miraheze/ssl · GitHub | github.com [16:26:42] [02ssl] 07AgentIsai closed pull request 03#595: Unset redirect for polandballwiki.com - 13https://github.com/miraheze/ssl/pull/595 [16:26:42] [url] Page not found · GitHub · GitHub | github.com [16:26:46] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/989c6817d4d3...087c72961df2 [16:26:47] [url] Comparing 989c6817d4d3...087c72961df2 · miraheze/ssl · GitHub | github.com [16:26:48] [02miraheze/ssl] 07AgentIsai 03087c729 - Unset redirect (#595) [16:26:58] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03revert-595-patch-4 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/319b3bf6718a [16:27:00] [02miraheze/ssl] 07MacFan4000 03319b3bf - Revert "Unset redirect (#595)" [16:27:01] [02ssl] 07MacFan4000 created branch 03revert-595-patch-4 - 13https://github.com/miraheze/ssl [16:27:02] [url] Page not found · GitHub · GitHub | github.com [16:27:03] [02ssl] 07MacFan4000 opened pull request 03#596: Revert "Unset redirect for polandballwiki.com" - 13https://github.com/miraheze/ssl/pull/596 [16:27:03] [url] Page not found · GitHub · GitHub | github.com [16:27:12] [02ssl] 07MacFan4000 closed pull request 03#596: Revert "Unset redirect for polandballwiki.com" - 13https://github.com/miraheze/ssl/pull/596 [16:27:12] ... [16:27:13] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/087c72961df2...541fece71b87 [16:27:14] [url] Comparing 087c72961df2...541fece71b87 · miraheze/ssl · GitHub | github.com [16:27:15] [02miraheze/ssl] 07MacFan4000 03541fece - Revert "Unset redirect (#595)" (#596) [16:27:16] [02ssl] 07MacFan4000 deleted branch 03revert-595-patch-4 - 13https://github.com/miraheze/ssl [16:27:17] [url] Page not found · GitHub · GitHub | github.com [16:27:18] [02miraheze/ssl] 07MacFan4000 deleted branch 03revert-595-patch-4 [16:27:23] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:27:28] oops heh you commented at the same time that i merged [16:27:59] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 0.017 second response time [16:28:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.85, 11.53, 10.52 [16:28:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.21, 11.98, 10.58 [16:28:06] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:28:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.80, 3.68 [16:28:34] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.080 second response time [16:28:39] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13210 MB (34% inode=96%); [16:29:02] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:29:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.43, 4.18, 3.31 [16:29:45] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:00] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:30:00] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:30:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.33, 11.89, 10.77 [16:30:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.29, 11.64, 10.63 [16:30:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.28, 4.55, 3.96 [16:30:48] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.050 second response time [16:30:50] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:30:57] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.027 second response time [16:31:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:31:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.79, 3.26 [16:31:58] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:32:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.12, 11.18, 10.66 [16:32:03] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:32:11] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.086 second response time [16:32:12] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.043 second response time [16:32:13] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.031 second response time [16:32:13] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:17] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.027 second response time [16:33:22] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [16:33:28] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.025 second response time [16:33:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.51, 4.38, 3.52 [16:34:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.10, 12.04, 10.93 [16:34:14] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 3.096 second response time [16:34:38] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:57] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.055 second response time [16:35:12] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:36:05] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:36:09] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:28] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.037 second response time [16:36:38] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.066 second response time [16:37:06] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:37:10] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:37:18] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.019 second response time [16:37:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.56, 11.73, 10.98 [16:38:05] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.021 second response time [16:38:09] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.047 second response time [16:38:48] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [16:38:49] [02puppet] 07Universal-Omega closed pull request 03#2786: users: update display_name for sre and mediawiki - 13https://github.com/miraheze/puppet/pull/2786 [16:38:50] [url] Page not found · GitHub · GitHub | github.com [16:38:51] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:38:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bad521690f65...8911e8bed4c8 [16:38:53] [url] Comparing bad521690f65...8911e8bed4c8 · miraheze/puppet · GitHub | github.com [16:38:54] [02miraheze/puppet] 07Universal-Omega 038911e8b - users: update display_name for sre and mediawiki (#2786) [16:38:55] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [16:38:56] [url] Page not found · GitHub · GitHub | github.com [16:39:02] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.141 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:39:06] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:39:20] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.015 second response time [16:39:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.82, 3.60 [16:40:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.39, 11.12, 10.84 [16:40:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 11.75, 11.20 [16:40:39] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:53] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.073 second response time [16:41:21] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:27] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:41:37] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.038 second response time [16:41:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.65, 13.45, 11.76 [16:42:10] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:21] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:31] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.021 second response time [16:42:56] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:15] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:43:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.19, 3.39 [16:43:26] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.023 second response time [16:43:45] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:47] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.10, 12.39, 11.53 [16:44:07] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [16:44:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.036 second response time [16:44:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [16:44:58] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:45:41] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.087 second response time [16:45:50] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.170 second response time [16:46:24] [02dns] 07MacFan4000 opened pull request 03#334: add 2 zones - 13https://github.com/miraheze/dns/pull/334 [16:46:25] [url] Page not found · GitHub · GitHub | github.com [16:46:38] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.033 second response time [16:46:43] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.061 second response time [16:46:50] CosmicAlpha: can you please merge ^^ when you have a chance? [16:47:05] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.037 second response time [16:47:21] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:28] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:47:39] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:47:42] MacFan4000: sure. [16:47:44] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:01] [02dns] 07Universal-Omega closed pull request 03#334: add 2 zones - 13https://github.com/miraheze/dns/pull/334 [16:48:02] ... [16:48:03] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±0] 13https://github.com/miraheze/dns/compare/6e88f458cfa5...4db328259ea2 [16:48:03] [url] Comparing 6e88f458cfa5...4db328259ea2 · miraheze/dns · GitHub | github.com [16:48:04] [02miraheze/dns] 07MacFan4000 034db3282 - add 2 zones (#334) [16:48:37] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 4.25, 3.76 [16:49:25] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.015 second response time [16:49:40] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:08] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 26% [16:50:14] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:47] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [16:51:40] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.021 second response time [16:52:11] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [16:54:42] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.071 second response time [16:54:47] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.038 second response time [16:54:50] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/5c24780ec227 [16:54:52] [02miraheze/ssl] 07MacFan4000 035c24780 - add wiki.yermocapital.com [16:54:53] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [16:54:54] [url] Page not found · GitHub · GitHub | github.com [16:55:17] [02ssl] 07MacFan4000 opened pull request 03#597: add 2 domains - 13https://github.com/miraheze/ssl/pull/597 [16:55:18] ... [16:55:30] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:55:54] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:00] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:19] [02ssl] 07MacFan4000 edited pull request 03#597: add 4 domains - 13https://github.com/miraheze/ssl/pull/597 [16:56:19] [url] Page not found · GitHub · GitHub | github.com [16:56:52] [02ssl] 07MacFan4000 edited pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [16:56:53] [url] Page not found · GitHub · GitHub | github.com [16:57:34] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:58:03] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.084 second response time [16:58:22] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.067 second response time [16:58:27] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:53] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:59:04] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/5c24780ec227...c98143d21cf8 [16:59:05] [url] Comparing 5c24780ec227...c98143d21cf8 · miraheze/ssl · GitHub | github.com [16:59:06] [02miraheze/ssl] 07MacFan4000 03c98143d - add psychevos.org [16:59:07] [02ssl] 07MacFan4000 synchronize pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [16:59:08] [url] Page not found · GitHub · GitHub | github.com [16:59:30] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:59:37] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:00:02] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:00:03] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005706250668 secs [17:00:26] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.022 second response time [17:00:56] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/c98143d21cf8...eb61d9a155ea [17:00:56] [url] Comparing c98143d21cf8...eb61d9a155ea · miraheze/ssl · GitHub | github.com [17:00:57] [02miraheze/ssl] 07MacFan4000 03eb61d9a - add removededm.ml [17:00:58] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [17:00:59] [02ssl] 07MacFan4000 synchronize pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [17:00:59] [url] Page not found · GitHub · GitHub | github.com [17:01:25] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:28] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:30] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:32] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:33] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:53] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:01:56] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:02:37] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:02:45] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:02:56] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/eb61d9a155ea...bd899ca8ef64 [17:02:56] [url] Comparing eb61d9a155ea...bd899ca8ef64 · miraheze/ssl · GitHub | github.com [17:02:57] [02miraheze/ssl] 07MacFan4000 03bd899ca - add dlfm-wiki.top [17:02:59] [02ssl] 07MacFan4000 synchronize pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [17:02:59] [url] Page not found · GitHub · GitHub | github.com [17:03:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.08, 0.04, 0.00 [17:04:16] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.018 second response time [17:04:34] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.07, 0.07 [17:04:36] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.162 second response time [17:05:07] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bd899ca8ef64...6b3bdc638fc9 [17:05:07] [url] Comparing bd899ca8ef64...6b3bdc638fc9 · miraheze/ssl · GitHub | github.com [17:05:08] [02miraheze/ssl] 07MacFan4000 036b3bdc6 - Update redirects.yaml [17:05:10] [02ssl] 07MacFan4000 synchronize pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [17:05:10] [url] Page not found · GitHub · GitHub | github.com [17:05:31] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.016 second response time [17:06:08] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:06:22] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:40] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:06:42] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -4.39286232e-05 secs [17:06:46] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.037 second response time [17:06:52] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [17:06:54] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:06:57] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [17:07:12] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.029 second response time [17:07:14] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:19] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 19% [17:08:22] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.037 second response time [17:08:28] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:38] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:39] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.016 second response time [17:08:51] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005502700806 secs [17:08:56] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6b3bdc638fc9...bdcc66168a40 [17:08:56] [url] Comparing 6b3bdc638fc9...bdcc66168a40 · miraheze/ssl · GitHub | github.com [17:08:57] [02miraheze/ssl] 07MacFan4000 03bdcc661 - Update certs.yaml [17:08:59] [02ssl] 07MacFan4000 synchronize pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [17:08:59] [url] Page not found · GitHub · GitHub | github.com [17:09:03] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:09:17] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 7.223 second response time [17:09:36] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:11] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 0.036 second response time [17:10:50] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 7.666 second response time [17:10:51] [02ssl] 07MacFan4000 closed pull request 03#597: add 4 domains + 1 redirect - 13https://github.com/miraheze/ssl/pull/597 [17:10:52] [url] Page not found · GitHub · GitHub | github.com [17:10:53] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+4/-0/±2] 13https://github.com/miraheze/ssl/compare/541fece71b87...2c8322ad1d7e [17:10:53] [url] Comparing 541fece71b87...2c8322ad1d7e · miraheze/ssl · GitHub | github.com [17:10:54] [02miraheze/ssl] 07MacFan4000 032c8322a - add 4 domains + 1 redirect (#597) [17:10:56] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:10:57] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:10:58] [url] Page not found · GitHub · GitHub | github.com [17:11:01] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:11:19] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.109 second response time [17:12:59] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.015 second response time [17:13:08] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.099 second response time [17:14:22] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:14:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:14:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:37] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [17:14:37] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.015 second response time [17:15:30] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [17:15:33] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:15:34] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:15:38] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:16:18] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 0.017 second response time [17:16:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.46, 3.96 [17:16:59] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:17:09] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [17:17:29] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:17:35] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.043 second response time [17:17:58] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:09] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:12] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.164 second response time [17:19:02] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:28] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:48] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.046 second response time [17:19:49] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:56] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13800 MB (35% inode=97%); [17:20:08] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:20] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.26, 3.01, 3.85 [17:21:03] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.747 second response time [17:21:22] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.014 second response time [17:21:26] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3346 bytes in 3.052 second response time [17:21:29] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:03] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:42] RECOVERY - gluster102 APT on gluster102 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [17:24:45] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:45] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.031 second response time [17:24:46] RECOVERY - es141 APT on es141 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [17:24:49] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:24:51] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.267 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:24:51] RECOVERY - es131 APT on es131 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [17:24:52] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [17:25:03] RECOVERY - db101 APT on db101 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [17:25:09] RECOVERY - gluster122 APT on gluster122 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [17:25:12] RECOVERY - mem101 APT on mem101 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [17:25:15] RECOVERY - db121 APT on db121 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [17:25:17] RECOVERY - cp31 APT on cp31 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [17:25:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 3.94, 3.84 [17:25:26] RECOVERY - ns1 APT on ns1 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [17:25:33] RECOVERY - db112 APT on db112 is OK: APT OK: 23 packages available for upgrade (0 critical updates). [17:25:38] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.034 second response time [17:25:42] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.029 second response time [17:25:51] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:05] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:20] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:26:24] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:32] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 7.154 second response time [17:26:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.41, 3.68 [17:27:28] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:38] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:27:42] RECOVERY - pwsc.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwsc.polandballwiki.com' will expire on Sun 02 Oct 2022 10:41:20 GMT +0000. [17:28:06] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.059 second response time [17:28:14] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [17:28:17] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.025 second response time [17:28:22] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.098 second response time [17:28:54] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 24% [17:28:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.42, 3.67 [17:29:16] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:26] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.018 second response time [17:29:27] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:42] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.047 second response time [17:29:51] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:30:17] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:01] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.68, 3.66, 3.80 [17:31:14] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.028 second response time [17:31:18] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:18] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13206 MB (34% inode=96%); [17:31:24] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [17:31:47] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:53] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.095 second response time [17:31:58] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:32:29] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:32:37] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.085 second response time [17:33:16] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:27] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:28] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:39] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:56] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:09] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.73, 1.92 [17:34:25] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.144 second response time [17:34:42] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.37, 2.43, 3.18 [17:34:53] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:35:28] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [17:36:19] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.016 second response time [17:36:21] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [17:36:44] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -3.388524055e-05 secs [17:37:13] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:30] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:35] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 3.033 second response time [17:38:05] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.042 second response time [17:38:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.21, 2.00, 1.98 [17:38:33] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.086 second response time [17:38:46] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:38:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 2.75, 3.30 [17:39:26] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:40:05] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.333 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:40:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.42, 1.79, 1.90 [17:40:19] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.026 second response time [17:40:39] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.033 second response time [17:40:58] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.038 second response time [17:41:28] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.109 second response time [17:42:02] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:42:11] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:42:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.28, 4.04, 3.44 [17:42:47] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.110 second response time [17:42:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 4.11, 3.77 [17:43:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3347 bytes in 1.031 second response time [17:43:46] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [17:43:46] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:44:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 6.92, 3.53, 2.50 [17:44:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.97, 3.75 [17:45:21] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:45:52] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 23% [17:45:57] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:46:19] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:46:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.42, 4.21, 3.84 [17:47:10] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.024 second response time [17:47:12] RECOVERY - polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - polandballwiki.com reverse DNS resolves to cp31.miraheze.org - CNAME FLAT [17:47:37] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:37] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:47:40] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:47:43] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:45] PROBLEM - www.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.polandballwiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:48:07] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13799 MB (35% inode=97%); [17:48:07] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.015 second response time [17:48:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.63, 3.67 [17:48:55] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:00] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:58] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.067 second response time [17:50:07] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:50:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.94, 3.70 [17:50:57] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:59] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.019 second response time [17:51:43] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.01, 0.00 [17:52:20] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:24] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.89, 3.70 [17:52:30] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.81, 3.69 [17:52:37] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.030 second response time [17:52:37] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:39] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:53] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:57] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3352 bytes in 3.050 second response time [17:53:18] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.238 second response time [17:53:30] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:53:48] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:54:09] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:54:26] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [17:54:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.78, 3.70 [17:54:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.15, 3.34, 3.53 [17:54:49] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.017 second response time [17:55:10] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.037 second response time [17:55:33] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.016 second response time [17:55:58] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [17:56:09] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.033 second response time [17:56:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.69, 4.09, 3.84 [17:56:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.56, 3.95, 3.69 [17:56:48] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:58] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:57:17] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [17:57:17] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:24] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [17:57:40] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.015 second response time [17:57:45] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.139 second response time [17:57:57] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [17:58:07] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:58:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.91, 3.81 [17:58:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.53, 3.57 [17:58:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:03] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:59:03] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:59:25] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:35] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:00:02] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:00:29] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 0.454 second response time [18:01:05] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:07] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 1.017 second response time [18:01:15] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [18:01:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:25] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.144 second response time [18:01:35] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:01:43] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.72, 2.88, 3.31 [18:02:37] RECOVERY - small.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'small.polandballwiki.com' will expire on Mon 10 Oct 2022 05:55:10 GMT +0000. [18:02:55] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:58] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.034 second response time [18:03:02] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:03:19] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:03:44] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:57] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:04:04] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.290 second response time [18:04:41] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:54] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:03] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.038 second response time [18:05:31] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:56] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.040 second response time [18:06:12] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.034 second response time [18:06:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.45, 3.52 [18:06:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.24, 3.90, 3.76 [18:06:34] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13798 MB (35% inode=97%); [18:06:41] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.050 second response time [18:06:53] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 25% [18:07:07] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.269 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:07:30] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:48] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:59] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:08:06] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.161 second response time [18:08:06] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [18:08:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.50, 4.33, 3.83 [18:08:15] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:19] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0005609095097 secs [18:08:30] RECOVERY - www.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - www.polandballwiki.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [18:08:35] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:08:48] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [18:08:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:09:14] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:26] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [18:09:46] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:50] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [18:09:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:10:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.87, 3.72 [18:10:16] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:10:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.27, 3.55 [18:10:20] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.024 second response time [18:10:25] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0007758438587 secs [18:10:37] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.014 second response time [18:11:05] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:11:21] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.032 second response time [18:11:54] RECOVERY - staff.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - staff.polandballwiki.com reverse DNS resolves to cp31.miraheze.org - CNAME FLAT [18:12:07] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:12:19] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.068 second response time [18:12:24] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:13:05] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 3.041 second response time [18:13:07] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:13:58] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.170 second response time [18:14:00] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [18:14:02] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:04] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.52, 3.56 [18:14:06] RECOVERY - fanon.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanon.polandballwiki.com' will expire on Tue 25 Oct 2022 07:14:58 GMT +0000. [18:14:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.542 second response time [18:14:15] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.64, 3.59 [18:14:24] RECOVERY - pwsc.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - pwsc.polandballwiki.com reverse DNS resolves to cp31.miraheze.org - CNAME FLAT [18:14:32] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [18:14:36] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:36] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.077 second response time [18:15:20] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 36987 bytes in 0.034 second response time [18:15:41] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:15:53] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:16:18] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [18:16:30] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.021 second response time [18:16:34] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.336 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:16:42] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13203 MB (34% inode=96%); [18:17:08] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:17:09] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:17:28] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.035 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:17:44] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.058 second response time [18:17:55] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [18:18:05] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3329 bytes in 1.044 second response time [18:18:27] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:28] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:18:32] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:37] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:39] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:16] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:31] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:34] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:35] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:20:10] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:20:23] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.063 second response time [18:20:27] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:21:38] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:49] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:21:53] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:21:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.61, 3.69 [18:22:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.55, 3.37, 3.54 [18:22:29] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.014 second response time [18:22:31] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.017 second response time [18:22:37] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.046 second response time [18:22:49] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.175 second response time [18:23:55] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.094 second response time [18:24:17] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 3.050 second response time [18:24:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 2.033 second response time [18:25:01] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13796 MB (35% inode=97%); [18:25:39] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:26:20] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:27:29] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:27:34] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [18:27:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:27:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.93, 4.14, 3.79 [18:28:03] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:06] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:09] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:28:18] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.03, 0.13, 0.06 [18:28:40] RECOVERY - small.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - small.polandballwiki.com reverse DNS resolves to cp30.miraheze.org - CNAME FLAT [18:28:45] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:56] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.37, 2.02, 1.51 [18:29:00] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:29:19] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:29:50] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [18:29:52] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:30:13] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.028 second response time [18:30:17] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:30:35] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:30:41] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:30:42] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:30:56] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:08] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:17] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 3.057 second response time [18:31:50] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:18] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0006615519524 secs [18:32:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 4.05, 3.71 [18:32:31] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.142 second response time [18:32:45] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0006230771542 secs [18:33:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3331 bytes in 1.035 second response time [18:33:21] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 20% [18:33:45] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:58] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:19] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:24] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:43] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:59] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.01, 0.00 [18:35:30] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [18:35:54] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:35:58] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:01] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.019 second response time [18:36:02] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 0.016 second response time [18:36:02] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [18:36:02] RECOVERY - staff.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'staff.polandballwiki.com' will expire on Sun 02 Oct 2022 10:43:17 GMT +0000. [18:36:06] RECOVERY - pt.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - pt.polandballwiki.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [18:36:09] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.97, 3.48, 3.63 [18:36:23] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.019 second response time [18:36:25] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:41] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.056 second response time [18:36:42] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.080 second response time [18:36:56] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:01] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [18:37:17] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:37:21] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:37:30] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.231 second response time [18:37:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.89, 3.94 [18:37:42] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:47] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37025 bytes in 7.244 second response time [18:37:55] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:37:56] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.042 second response time [18:38:06] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.051 second response time [18:38:17] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 2.042 second response time [18:38:22] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:38:40] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13200 MB (34% inode=96%); [18:38:51] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:38:56] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.61, 1.93, 1.75 [18:39:23] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.073 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:39:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.87, 4.10, 4.02 [18:39:39] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:00] RECOVERY - commons.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.polandballwiki.com' will expire on Mon 10 Oct 2022 05:30:54 GMT +0000. [18:40:13] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 2.90, 3.37 [18:41:11] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.125 second response time [18:41:18] RECOVERY - commons.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - commons.polandballwiki.com reverse DNS resolves to cp31.miraheze.org - CNAME OK [18:41:37] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 19% [18:41:51] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0006346404552 secs [18:41:53] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3343 bytes in 3.050 second response time [18:42:09] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:42:15] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.022 second response time [18:42:24] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:25] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:42:27] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.04, 0.13, 0.08 [18:42:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:43:00] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:43:24] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:24] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.331 second response time [18:43:56] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:44:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.09, 4.19, 3.79 [18:44:21] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14726 bytes in 0.017 second response time [18:44:23] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [18:44:27] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.054 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:44:30] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:44:56] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 0.44, 1.25, 1.53 [18:44:57] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.410 second response time [18:45:17] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [18:45:38] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:46:07] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13793 MB (35% inode=97%); [18:46:20] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:23] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [18:47:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 3.055 second response time [18:48:31] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:49:08] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 36987 bytes in 1.070 second response time [18:49:12] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:49] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.087 second response time [18:49:59] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:00] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:12] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:21] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:21] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:25] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.054 second response time [18:50:33] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 20% [18:51:18] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:51:33] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:00] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:02] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13199 MB (34% inode=96%); [18:52:22] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 1.292 second response time [18:52:53] RECOVERY - fanon.polandballwiki.com - reverse DNS on sslhost is OK: SSL OK - fanon.polandballwiki.com reverse DNS resolves to cp30.miraheze.org - CNAME OK [18:53:01] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:53:39] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:54:01] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 5.137 second response time [18:55:09] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:55:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.126 second response time [18:55:18] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 30 minutes ago with 0 failures [18:55:26] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.022 second response time [18:55:55] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:55:56] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:56:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:01] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [18:57:03] PROBLEM - graylog121 APT on graylog121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:57:16] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:57] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:58:27] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.17, 0.15, 0.06 [18:58:38] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:04] PROBLEM - graylog121 APT on graylog121 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (4 critical updates). [18:59:19] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.203 second response time [19:00:36] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:00:53] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:26] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:01:39] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:40] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:01:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:02:29] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [19:03:01] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:03:02] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.079 second response time [19:03:03] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:34] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.033 second response time [19:03:48] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:03:50] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.018 second response time [19:03:52] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.026 second response time [19:05:01] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3343 bytes in 1.044 second response time [19:05:07] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 13197 MB (34% inode=96%); [19:05:14] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:05:35] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:01] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:34] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:49] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:51] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 7.448 second response time [19:07:05] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 1.948 second response time [19:07:20] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:32] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:47] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.020 second response time [19:08:10] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:08:54] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.013 second response time [19:09:12] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:09:12] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:09:23] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.020 second response time [19:09:29] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:09:32] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:09:57] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 0.017 second response time [19:10:04] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:10:23] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:24] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:10:27] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:10:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:11:01] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:11] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [19:11:20] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:11:26] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.020 second response time [19:11:36] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.01, 0.02, 0.00 [19:11:59] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.111 second response time [19:12:16] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:12:20] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.016 second response time [19:12:21] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:28] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [19:12:29] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:41] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:39] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:13:42] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 21% [19:13:50] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:13:52] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:14] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:14:22] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 3.224 second response time [19:14:26] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3343 bytes in 0.017 second response time [19:14:37] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:55] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.061 second response time [19:14:55] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw121 [19:15:03] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:15:17] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.59, 8.69, 7.47 [19:15:18] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:15:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29521 bytes in 1.109 second response time [19:15:39] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 1.027 second response time [19:15:43] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:15:48] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.018 second response time [19:15:49] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.593 second response time [19:16:05] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.079 second response time [19:16:12] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.087 second response time [19:16:13] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.034 second response time [19:16:30] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37033 bytes in 0.034 second response time [19:16:45] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.043 second response time [19:16:48] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:16:54] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:17:13] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.021 second response time [19:17:16] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.17, 8.23, 7.45 [19:17:45] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 5.098 second response time [19:18:02] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:19:08] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 1.254 second response time [19:19:28] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 8.999 second response time [19:19:37] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:20:02] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:20:16] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:20:19] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [19:20:33] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.360 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:20:44] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/8911e8bed4c8...40f1c1e180c6 [19:20:45] [url] Comparing 8911e8bed4c8...40f1c1e180c6 · miraheze/puppet · GitHub | github.com [19:20:46] [02miraheze/puppet] 07JohnFLewis 0340f1c1e - dns: add latency to prom export [19:21:12] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:21:18] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:21:25] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:21:59] PROBLEM - cp31 APT on cp31 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:22:01] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:03] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 26 minutes ago with 0 failures [19:22:37] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:23:03] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:24:01] RECOVERY - cp31 APT on cp31 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:24:58] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:03] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:25:47] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.068 second response time [19:26:01] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.030 second response time [19:26:02] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:26:10] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 1.035 second response time [19:26:11] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 100174 bytes in 0.888 second response time [19:26:20] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:26:44] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:26:52] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:26:57] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3343 bytes in 1.050 second response time [19:27:23] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [19:27:30] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:27:35] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 1.037 second response time [19:27:48] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:11] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:28:21] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:28:41] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:29:33] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:29:58] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:30:22] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:32] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.27, 0.14, 0.04 [19:30:36] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.124 second response time [19:30:41] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:42] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 13790 MB (35% inode=97%); [19:31:26] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.039 second response time [19:32:01] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.100 second response time [19:32:18] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:32:21] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [19:32:39] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:32:55] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:33:03] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:33:18] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [19:33:23] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.018 second response time [19:33:51] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14742 bytes in 2.039 second response time [19:34:06] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:34:24] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:05] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 5.070 second response time [19:35:06] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [19:35:08] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:35:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:28] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 7.315 second response time [19:35:59] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.244 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:36:15] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/40f1c1e180c6...b8527546f94e [19:36:15] [url] Comparing 40f1c1e180c6...b8527546f94e · miraheze/puppet · GitHub | github.com [19:36:16] [02miraheze/puppet] 07JohnFLewis 03b852754 - prometheus: fix python variable [19:36:23] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3344 bytes in 1.034 second response time [19:36:43] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:01] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:04] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.019 second response time [19:37:58] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [19:37:59] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.264 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:38:26] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:38:53] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:39:39] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:39:53] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 20% [19:39:53] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:56] !log downtime cp20 and cp21 in icinga, OVH investigating problems we can't control [19:41:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:00] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:44:08] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [19:45:48] JohnLewis: how does icinga ldap work? Because I can still login but it shows nothing [19:46:07] * RhinosF1 once mixed it up for Fossbots icinga and wondered why no dashboards [19:46:26] My ldap is still active in case any useful mail came [19:49:28] Off the top of the head, no idea [19:54:38] all ldap accounts can login, I think there's no default view group though unless logging in as guest user [19:58:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:04:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.12, 3.87 [20:05:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.80, 2.79, 3.81 [20:05:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:07:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.12, 3.69, 3.98 [20:09:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.35, 3.83 [20:11:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.57, 3.63, 3.84 [20:13:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.47, 3.74 [20:19:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.31, 2.41, 3.25 [20:20:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 3.04, 3.40 [20:31:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 9.35, 11.77 [20:35:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 10.05, 11.92 [20:37:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.05, 11.72, 12.27 [20:37:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.38, 11.05, 11.75 [20:39:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 10.53, 11.48 [20:40:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.88, 3.44 [20:43:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 10.52, 11.65 [20:44:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.24, 3.34, 3.33 [20:47:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.70, 3.20 [20:47:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:49:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 3.19, 3.07 [20:51:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.22, 10.81, 11.20 [20:53:21] PROBLEM - cp30 APT on cp30 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [20:53:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.36, 10.31, 10.96 [20:55:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.46, 3.97, 3.54 [20:55:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.63, 11.16, 11.19 [20:55:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.94, 4.08, 3.37 [20:55:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.12, 11.23, 10.85 [20:57:40] RECOVERY - cp30 APT on cp30 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [20:57:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.38, 3.20 [20:59:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.94, 3.63 [20:59:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 11.62, 11.12 [21:00:53] PROBLEM - wiki.zergmafia.top - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zergmafia.top' expires in 15 day(s) (Tue 06 Sep 2022 20:57:49 GMT +0000). [21:01:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2c8322ad1d7e...9e2ef16be58b [21:01:17] [url] Comparing 2c8322ad1d7e...9e2ef16be58b · miraheze/ssl · GitHub | github.com [21:01:18] [02miraheze/ssl] 07MirahezeSSLBot 039e2ef16 - Bot: Update SSL cert for wiki.zergmafia.top [21:01:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 4.22, 3.62 [21:01:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.48, 11.88, 11.61 [21:03:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.75, 12.00, 11.67 [21:03:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.65, 13.33, 11.78 [21:03:48] PROBLEM - wiki.pixlies.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pixlies.net' expires in 15 day(s) (Tue 06 Sep 2022 20:56:25 GMT +0000). [21:03:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:05:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.76, 3.60 [21:05:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.00, 11.30, 11.44 [21:06:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 2.99, 3.32 [21:08:54] PROBLEM - encyclopediarobotica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediarobotica.org' expires in 15 day(s) (Tue 06 Sep 2022 20:51:36 GMT +0000). [21:09:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9e2ef16be58b...b35f4533e5dc [21:09:24] [url] Comparing 9e2ef16be58b...b35f4533e5dc · miraheze/ssl · GitHub | github.com [21:09:25] [02miraheze/ssl] 07MirahezeSSLBot 03b35f453 - Bot: Update SSL cert for wiki.pixlies.net [21:09:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.81, 12.07, 11.62 [21:09:59] PROBLEM - wikifencing.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikifencing.ru' expires in 15 day(s) (Tue 06 Sep 2022 20:55:14 GMT +0000). [21:11:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.21, 3.40 [21:11:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b35f4533e5dc...ee3f0caef8fe [21:11:33] [url] Comparing b35f4533e5dc...ee3f0caef8fe · miraheze/ssl · GitHub | github.com [21:11:34] [02miraheze/ssl] 07MirahezeSSLBot 03ee3f0ca - Bot: Update SSL cert for encyclopediarobotica.org [21:12:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ee3f0caef8fe...d6d6116f97d9 [21:12:36] [url] Comparing ee3f0caef8fe...d6d6116f97d9 · miraheze/ssl · GitHub | github.com [21:12:36] [02miraheze/ssl] 07MirahezeSSLBot 03d6d6116 - Bot: Update SSL cert for wikifencing.ru [21:12:49] PROBLEM - wildbowiki.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wildbowiki.net' expires in 15 day(s) (Tue 06 Sep 2022 20:59:02 GMT +0000). [21:14:47] PROBLEM - nl.xliving.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nl.xliving.org' expires in 15 day(s) (Tue 06 Sep 2022 20:54:18 GMT +0000). [21:17:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.88, 3.62 [21:17:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 11.47, 11.86 [21:19:10] PROBLEM - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pbptanarchy.tk' expires in 15 day(s) (Tue 06 Sep 2022 20:52:59 GMT +0000). [21:19:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.59, 3.53 [21:19:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d6d6116f97d9...846a4ce31a4d [21:19:56] [url] Comparing d6d6116f97d9...846a4ce31a4d · miraheze/ssl · GitHub | github.com [21:19:57] [02miraheze/ssl] 07MirahezeSSLBot 03846a4ce - Bot: Update SSL cert for wildbowiki.net [21:21:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 4.01, 3.68 [21:22:10] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Tue 06 Sep 2022 21:18:17 GMT +0000). [21:23:14] PROBLEM - wiki.wisrail.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wisrail.de' expires in 15 day(s) (Tue 06 Sep 2022 21:01:07 GMT +0000). [21:23:44] PROBLEM - wiki.lu9.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lu9.io' expires in 15 day(s) (Tue 06 Sep 2022 20:59:54 GMT +0000). [21:25:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.98, 11.23, 12.00 [21:26:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/846a4ce31a4d...503a97147a92 [21:26:33] [url] Comparing 846a4ce31a4d...503a97147a92 · miraheze/ssl · GitHub | github.com [21:26:34] [02miraheze/ssl] 07MirahezeSSLBot 03503a971 - Bot: Update SSL cert for wiki.pbptanarchy.tk [21:27:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.34, 13.89, 12.86 [21:28:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.43, 3.05 [21:29:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.43, 8.83, 10.15 [21:30:06] RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zergmafia.top' will expire on Sat 19 Nov 2022 20:01:10 GMT +0000. [21:30:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.18, 3.02 [21:31:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.98, 3.91 [21:32:10] PROBLEM - housing.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'housing.wiki' expires in 15 day(s) (Tue 06 Sep 2022 21:04:59 GMT +0000). [21:33:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/503a97147a92...5a3512e0099c [21:33:32] [url] Comparing 503a97147a92...5a3512e0099c · miraheze/ssl · GitHub | github.com [21:33:32] RECOVERY - wiki.pixlies.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pixlies.net' will expire on Sat 19 Nov 2022 20:09:18 GMT +0000. [21:33:33] [02miraheze/ssl] 07MirahezeSSLBot 035a3512e - Bot: Update SSL cert for nl.xliving.org [21:33:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.82, 10.62, 10.61 [21:35:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:37:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.45, 11.81, 10.93 [21:37:50] RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Sat 19 Nov 2022 20:11:27 GMT +0000. [21:38:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a3512e0099c...95d9bd3d5ca0 [21:38:15] [url] Comparing 5a3512e0099c...95d9bd3d5ca0 · miraheze/ssl · GitHub | github.com [21:38:16] [02miraheze/ssl] 07MirahezeSSLBot 0395d9bd3 - Bot: Update SSL cert for wiki.wisrail.de [21:39:06] RECOVERY - wikifencing.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wikifencing.ru' will expire on Sat 19 Nov 2022 20:12:29 GMT +0000. [21:39:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.01, 2.64, 3.29 [21:42:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/95d9bd3d5ca0...8a5213739f1f [21:42:05] [url] Comparing 95d9bd3d5ca0...8a5213739f1f · miraheze/ssl · GitHub | github.com [21:42:06] [02miraheze/ssl] 07MirahezeSSLBot 038a52137 - Bot: Update SSL cert for housing.wiki [21:45:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8a5213739f1f...531face25e0e [21:45:46] [url] Comparing 8a5213739f1f...531face25e0e · miraheze/ssl · GitHub | github.com [21:45:47] [02miraheze/ssl] 07MirahezeSSLBot 03531face - Bot: Update SSL cert for wiki.lu9.io [21:52:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.56, 4.66, 3.84 [21:53:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.08, 11.51, 11.70 [21:57:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 4.18, 3.38 [21:59:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.47, 3.54, 3.24 [22:00:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.49, 3.72 [22:01:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.41, 2.75, 2.98 [22:01:07] RECOVERY - housing.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'housing.wiki' will expire on Sat 19 Nov 2022 20:41:59 GMT +0000. [22:01:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.41, 10.19, 11.85 [22:03:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.16, 8.50, 10.04 [22:03:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [22:04:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 2.85, 3.39 [22:10:49] RECOVERY - wildbowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wildbowiki.net' will expire on Sat 19 Nov 2022 20:19:51 GMT +0000. [22:13:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.32, 8.07, 9.99 [22:13:53] RECOVERY - nl.xliving.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nl.xliving.org' will expire on Sat 19 Nov 2022 20:33:25 GMT +0000. [22:15:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 4.37, 3.59 [22:17:16] RECOVERY - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pbptanarchy.tk' will expire on Sat 19 Nov 2022 20:26:27 GMT +0000. [22:17:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 3.39, 3.32 [22:20:38] PROBLEM - eng.archiopedia.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns179.a.register.com.', 'dns025.c.register.com.', 'dns032.b.register.com.', 'dns083.d.register.com.'], 'CNAME': None} [22:21:14] RECOVERY - wiki.wisrail.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wisrail.de' will expire on Sat 19 Nov 2022 20:38:09 GMT +0000. [22:21:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:22:50] RECOVERY - wiki.lu9.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lu9.io' will expire on Sat 19 Nov 2022 20:45:40 GMT +0000. [22:29:39] [02miraheze/TSPortal] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/commit/443bccd108ed [22:29:40] [02miraheze/TSPortal] 07Universal-Omega 03443bccd - Update README [22:29:42] [02TSPortal] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/TSPortal [22:29:42] [url] Page not found · GitHub · GitHub | github.com [22:29:43] [02TSPortal] 07Universal-Omega opened pull request 03#6: Update README - 13https://github.com/miraheze/TSPortal/pull/6 [22:29:44] [url] Page not found · GitHub · GitHub | github.com [22:31:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 2.73, 4.71, 3.92 [22:33:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.96, 3.74 [22:35:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 10.82, 10.25 [22:37:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.46, 9.03, 9.67 [22:41:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.25, 3.83 [22:43:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.37, 3.55 [22:47:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 2.95, 3.33 [22:52:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.63, 9.87, 9.10 [22:54:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.60, 9.97, 9.25 [22:58:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 11.73, 10.14 [23:00:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.25, 11.51, 10.24 [23:03:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.30, 12.01, 10.67 [23:05:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 11.04, 10.49 [23:12:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.55, 11.09, 9.71 [23:14:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 10.33, 9.59 [23:15:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.93, 12.10, 11.35 [23:16:06] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.82, 9.51, 9.41 [23:17:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.26, 10.26, 10.77 [23:18:53] RECOVERY - eng.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - eng.archiopedia.org reverse DNS resolves to cp31.miraheze.org - CNAME OK [23:28:31] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4221 MB (10% inode=96%); [23:29:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.36, 9.54, 10.13 [23:33:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.42, 10.53, 10.43 [23:38:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 9.61, 8.89 [23:42:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.16, 9.96, 9.23 [23:47:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.51, 9.65, 10.16 [23:50:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.38, 10.19, 9.44 [23:50:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.00, 2.34 [23:51:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.23, 10.57, 10.46 [23:52:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.57, 11.16, 9.87 [23:52:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.56, 2.67 [23:53:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.35, 11.36, 10.75 [23:54:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.44, 10.92, 9.94 [23:54:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.04, 3.22, 2.65 [23:55:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 11.41, 10.89 [23:56:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.94, 11.16, 10.12 [23:56:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.99, 2.61 [23:58:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.48, 4.65, 3.29