[00:00:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.82, 2.34 [00:00:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.83, 1.54 [00:01:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 10.54, 8.49 [00:02:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.91, 3.26, 3.65 [00:02:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.60, 11.12, 10.16 [00:03:01] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:03] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 7219 MB (18% inode=96%); [00:03:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:09] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:23] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:36] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 0.23, 2.10, 4.59 [00:03:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.04, 7.62, 7.67 [00:03:59] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [00:04:02] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:02] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.32, 2.20, 3.21 [00:04:06] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:10] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:12] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:15] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:16] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 56% [00:04:30] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:31] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:35] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:36] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:37] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 2.06, 8.00, 9.13 [00:04:38] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:40] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:04:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.60, 1.54 [00:04:43] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:46] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:49] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:51] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [00:04:52] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:56] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:04:57] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:58] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:59] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 8563 MB (22% inode=96%); [00:05:00] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:01] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:02] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:02] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:02] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:03] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:06] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:11] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:05:14] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:17] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:17] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:19] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:21] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:29] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:59] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 55% [00:06:14] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:06:38] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [00:06:46] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 41% [00:06:57] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.717 second response time [00:06:58] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.008 second response time [00:07:01] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.783 second response time [00:07:01] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.718 second response time [00:07:04] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.408 second response time [00:07:05] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.195 second response time [00:07:10] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.806 second response time [00:07:14] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.336 second response time [00:07:15] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.329 second response time [00:07:15] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.308 second response time [00:07:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [00:07:18] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.313 second response time [00:07:19] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.331 second response time [00:07:23] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [00:07:28] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [00:07:59] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [00:08:07] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.308 second response time [00:08:08] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.215 second response time [00:08:09] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [00:08:12] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [00:08:13] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [00:08:14] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [00:08:19] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.006 second response time [00:08:28] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.007 second response time [00:08:34] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.006 second response time [00:08:35] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.213 second response time [00:08:37] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [00:08:38] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 4% [00:08:39] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [00:08:40] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [00:08:42] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 2% [00:08:42] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [00:08:45] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [00:08:49] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.067 second response time [00:08:50] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.302 second response time [00:08:56] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [00:08:57] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.042 second response time [00:08:59] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 3.017 second response time [00:08:59] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.969 second response time [00:09:00] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.724 second response time [00:09:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.765 second response time [00:09:08] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [00:09:11] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [00:09:59] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 5% [00:10:05] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 3% [00:11:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.56, 9.78, 8.08 [00:12:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.66, 3.93, 3.33 [00:13:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.29, 8.95, 8.01 [00:17:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.92, 12.98, 10.39 [00:17:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.42, 12.05, 9.55 [00:20:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 1.98, 1.53 [00:22:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.80, 1.52 [00:22:47] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.62, 4.15, 2.71 [00:22:49] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:32] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.68, 1.51 [00:24:42] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.60, 3.42, 2.60 [00:24:43] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [00:26:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 2.23, 3.96 [00:26:37] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.20, 3.16, 2.61 [00:28:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.39, 4.13, 4.41 [00:32:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 2.95, 3.83 [00:40:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 2.57, 3.29 [00:42:00] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.85, 4.12, 3.23 [00:43:55] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.24, 3.33, 3.05 [00:51:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 2.09, 1.65 [00:53:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.84, 1.61 [00:55:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.94, 1.66 [00:57:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.61, 1.58 [01:03:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.76, 1.66 [01:05:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.56, 1.60 [01:15:52] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.08, 1.62, 1.25 [01:16:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:43] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.46, 1.24 [01:18:18] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:28] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:25] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.688 second response time [01:22:32] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.135 second response time [01:22:36] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.028 second response time [01:23:04] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.551 second response time [01:23:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.38, 2.33, 1.87 [01:23:12] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 4.954 second response time [01:25:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.86, 1.75 [01:29:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.72, 1.95, 1.78 [01:29:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:27] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:21] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.30, 10.86, 7.92 [01:31:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.76, 1.73 [01:31:25] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 5.685 second response time [01:31:29] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.284 second response time [01:32:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 6.10, 9.30, 7.73 [01:33:10] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 2.00, 1.47, 1.29 [01:35:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.48, 2.27, 1.91 [01:35:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.10, 1.78, 1.40 [01:37:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.25, 1.48, 1.33 [01:37:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.87, 1.81 [01:45:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.13, 2.09, 1.89 [01:47:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.87, 1.84 [01:50:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 2.28, 2.02 [01:51:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.95, 2.02, 1.53 [01:53:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.72, 1.48 [01:55:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.91, 1.39, 1.39 [01:55:55] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.85, 3.85, 3.08 [01:56:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.30, 2.42 [01:56:50] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.73, 1.77, 1.92 [01:57:05] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:19] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.951 second response time [01:59:06] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.151 second response time [01:59:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 7.583 second response time [01:59:55] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.25, 3.72, 3.22 [02:00:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.57, 3.80, 2.93 [02:00:16] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.646 second response time [02:00:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.63, 1.80 [02:02:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.72, 3.21, 2.83 [02:04:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.05, 1.47, 1.71 [02:06:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.33, 1.63 [02:09:55] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.32, 3.64, 3.34 [02:11:55] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.63, 3.70, 3.41 [02:13:55] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.64, 3.17, 3.24 [02:21:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.57, 1.33 [02:23:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.64, 1.37 [02:25:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.42, 1.32 [02:26:11] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:54] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:36] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:07] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.344 second response time [02:28:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.27, 1.88, 1.10 [02:29:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.37, 3.11, 2.14 [02:30:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.44, 3.52, 2.93 [02:30:14] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.64, 1.44, 1.03 [02:30:55] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [02:30:56] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:59] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:35] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.028 second response time [02:32:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.75, 2.87, 2.76 [02:32:35] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.314 second response time [02:32:55] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.772 second response time [02:33:03] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.309 second response time [02:33:12] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 4.684 second response time [02:37:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.62, 1.42 [02:37:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.91, 1.79, 1.91 [02:39:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.38, 1.62, 1.45 [02:39:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.86, 1.92 [02:41:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.53, 1.79 [02:45:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.82, 2.28, 1.81 [02:45:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.82, 1.27, 1.63 [02:46:55] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.04, 3.89, 3.44 [02:47:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.81, 1.76, 1.68 [02:48:50] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.10, 3.80, 3.47 [02:51:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.34, 1.70, 1.68 [02:52:41] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.44, 3.33, 3.37 [02:55:46] !log delete puppet111 vm [02:55:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:53] [02dns] 07paladox closed pull request 03#325: Remove puppet111 from dns - 13https://github.com/miraheze/dns/pull/325 [02:56:53] [url] Page not found · GitHub · GitHub | github.com [02:56:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/106c76eddb19...cb5022f92e4d [02:56:55] [url] Comparing 106c76eddb19...cb5022f92e4d · miraheze/dns · GitHub | github.com [02:56:56] [02miraheze/dns] 07Universal-Omega 03cb5022f - Remove puppet111 from dns (#325) [03:03:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.03, 1.77, 1.75 [03:03:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.74, 1.66 [03:05:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.39, 1.61 [03:05:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.34, 1.52 [03:06:06] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:09] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:15] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:39] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.75, 6.39, 4.30 [03:07:12] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw121 mw131 mw141 mw142 [03:07:17] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw122 mw131 [03:07:25] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw122 mw131 [03:07:51] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 48% [03:09:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 2.09, 7.21, 11.89 [03:09:50] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 12% [03:10:02] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.335 second response time [03:10:07] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.199 second response time [03:10:12] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:14] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.331 second response time [03:10:16] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [03:10:19] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 44% [03:10:30] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [03:11:14] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [03:12:07] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [03:12:17] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 2% [03:12:24] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.87, 5.30, 4.61 [03:12:24] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 3% [03:13:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [03:13:07] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [03:14:18] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.98, 7.49, 5.48 [03:14:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.78, 9.53, 11.25 [03:15:33] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:16:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 8.90, 10.79 [03:17:34] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.311 second response time [03:18:08] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.56, 5.34, 5.09 [03:20:03] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.39, 4.85, 4.94 [03:20:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.94, 9.83, 11.73 [03:22:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.80, 7.80, 9.93 [03:23:55] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.50, 5.61, 5.19 [03:24:01] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [03:24:48] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 53% [03:24:54] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [03:24:54] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:07] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw141 mw142 [03:25:17] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [03:26:22] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [03:26:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.76, 6.61, 9.60 [03:26:48] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 16% [03:26:53] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 10% [03:27:57] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 28% [03:28:01] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:05] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:08] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:08] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:09] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:10] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:11] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:12] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:13] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:14] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:16] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:16] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:28:19] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:21] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [03:28:54] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:58] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:02] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:11] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [03:29:18] PROBLEM - cp31 Stunnel HTTP for matomo131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [03:30:02] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.541 second response time [03:30:04] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [03:30:04] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.319 second response time [03:30:05] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.331 second response time [03:30:05] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.306 second response time [03:30:07] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [03:30:07] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [03:30:08] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.202 second response time [03:30:09] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 1.219 second response time [03:30:10] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.313 second response time [03:30:11] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [03:30:14] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [03:30:14] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.477 second response time [03:30:15] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [03:30:22] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.210 second response time [03:30:51] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.209 second response time [03:30:56] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.102 second response time [03:31:00] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [03:31:07] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [03:31:13] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.720 second response time [03:31:55] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.17, 4.42, 4.96 [03:33:08] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [03:33:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.71, 13.36, 11.38 [03:37:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.07, 11.72, 11.15 [03:41:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.92, 13.39, 11.81 [03:41:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.73, 12.45, 10.38 [03:45:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.80, 1.55 [03:47:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 11.52, 10.79 [03:47:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.35, 1.60, 1.50 [03:51:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.69, 11.63, 10.89 [03:53:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.90, 11.71, 11.02 [03:55:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.35, 12.27, 11.30 [03:57:27] !log root@gluster122:/home/paladox# gluster volume remove-brick static gluster111.miraheze.org:/srv/static force (6 files failed to migrate though) [03:57:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.17, 11.05, 10.96 [03:58:37] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:00] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:24] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:25] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 3.103 second response time [03:59:29] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [03:59:37] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:39] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:55] PROBLEM - gluster111 glusterd Volume on gluster111 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfsd' [04:00:36] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [04:01:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.99, 8.49, 9.98 [04:03:23] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 9.642 second response time [04:03:52] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:00] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:00] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:17] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.31, 9.88, 11.72 [04:04:58] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:01] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:07] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:13] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw122 mw142 [04:05:28] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.877 second response time [04:05:29] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [04:05:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 1.123 second response time [04:05:41] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.051 second response time [04:05:46] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.341 second response time [04:05:52] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw121 [04:06:16] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [04:06:52] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.98, 2.16, 1.54 [04:06:53] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [04:07:02] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.757 second response time [04:07:03] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.601 second response time [04:07:29] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:07:31] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:07:32] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.509 second response time [04:07:46] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [04:07:47] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:54] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:55] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:00] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:01] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.104 second response time [04:09:21] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.089 second response time [04:09:27] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.334 second response time [04:09:30] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.868 second response time [04:09:32] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [04:09:45] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [04:09:54] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.268 second response time [04:09:55] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 2.492 second response time [04:09:55] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.861 second response time [04:10:00] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.330 second response time [04:10:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.53, 13.45, 12.52 [04:10:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.05, 1.86, 1.58 [04:11:13] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [04:12:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.62, 14.09, 11.30 [04:12:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.50, 1.48 [04:13:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/57df424d27dc...72ae361b8724 [04:13:46] [url] Comparing 57df424d27dc...72ae361b8724 · miraheze/mw-config · GitHub | github.com [04:13:47] [02miraheze/mw-config] 07Universal-Omega 0372ae361 - Add swift to disallowed subdomains [04:15:00] miraheze/mw-config - Universal-Omega the build passed. [04:18:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.77, 1.72, 1.59 [04:22:31] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.70, 1.62, 1.56 [04:27:11] !log [@test131] starting deploy of {'config': True} to all [04:27:12] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:27:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:27:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:28:00] RECOVERY - test131 Check Gluster Clients on test131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [04:35:56] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:00] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 9.595 second response time [04:38:50] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:54] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:40:07] !log [@mwtask141] starting deploy of {'config': True} to all [04:40:51] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 43s [04:40:51] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.489 second response time [04:40:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:52] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.593 second response time [04:40:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.52, 1.27 [04:43:05] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/4605026082e9 [04:43:06] [02miraheze/dns] 07paladox 034605026 - experiment with load balancing gluster [04:43:08] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [04:43:08] [url] Page not found · GitHub · GitHub | github.com [04:43:09] [02dns] 07paladox opened pull request 03#327: experiment with load balancing gluster - 13https://github.com/miraheze/dns/pull/327 [04:43:10] [url] Page not found · GitHub · GitHub | github.com [04:43:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.22, 1.19 [04:43:19] [02dns] 07paladox closed pull request 03#327: experiment with load balancing gluster - 13https://github.com/miraheze/dns/pull/327 [04:43:20] [url] Page not found · GitHub · GitHub | github.com [04:43:21] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cb5022f92e4d...9e2286ece7d4 [04:43:21] [url] Comparing cb5022f92e4d...9e2286ece7d4 · miraheze/dns · GitHub | github.com [04:43:22] [02miraheze/dns] 07paladox 039e2286e - experiment with load balancing gluster (#327) [04:44:24] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [04:44:26] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [04:44:26] [url] Page not found · GitHub · GitHub | github.com [04:45:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/be3594ceeaf1 [04:45:14] [02miraheze/puppet] 07paladox 03be3594c - mw121/22/31/32/41/44: switch gluster volume to use load balanced version [04:45:15] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [04:45:16] ... [04:45:17] [02puppet] 07paladox opened pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:45:17] [url] Page not found · GitHub · GitHub | github.com [04:45:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/be3594ceeaf1...c59e4db824f3 [04:45:35] [url] Comparing be3594ceeaf1...c59e4db824f3 · miraheze/puppet · GitHub | github.com [04:45:36] [02miraheze/puppet] 07paladox 03c59e4db - Update mw132.yaml [04:45:37] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:45:37] ... [04:45:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c59e4db824f3...c4c5b6f1e5e0 [04:45:45] [url] Comparing c59e4db824f3...c4c5b6f1e5e0 · miraheze/puppet · GitHub | github.com [04:45:45] [02miraheze/puppet] 07paladox 03c4c5b6f - Update mw141.yaml [04:45:47] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:45:47] [url] Page not found · GitHub · GitHub | github.com [04:45:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c4c5b6f1e5e0...5eaed387566c [04:45:56] [url] Comparing c4c5b6f1e5e0...5eaed387566c · miraheze/puppet · GitHub | github.com [04:45:57] [02miraheze/puppet] 07paladox 035eaed38 - Update mw142.yaml [04:45:58] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:45:59] [url] Page not found · GitHub · GitHub | github.com [04:46:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5eaed387566c...0daf9891fbda [04:46:17] [url] Comparing 5eaed387566c...0daf9891fbda · miraheze/puppet · GitHub | github.com [04:46:18] [02miraheze/puppet] 07paladox 030daf989 - Update mw121.yaml [04:46:20] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:46:20] [url] Page not found · GitHub · GitHub | github.com [04:47:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9e2286ece7d4...6484a6129355 [04:47:07] [url] Comparing 9e2286ece7d4...6484a6129355 · miraheze/dns · GitHub | github.com [04:47:08] [02miraheze/dns] 07paladox 036484a61 - Forgot to add the gluster weight to miraheze.org [04:48:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0daf9891fbda...07c4e8454ec8 [04:48:38] [url] Comparing 0daf9891fbda...07c4e8454ec8 · miraheze/puppet · GitHub | github.com [04:48:38] [02miraheze/puppet] 07paladox 0307c4e84 - Update mw122.yaml [04:48:40] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:48:40] [url] Page not found · GitHub · GitHub | github.com [04:48:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/07c4e8454ec8...28ea980c3709 [04:48:49] [url] Comparing 07c4e8454ec8...28ea980c3709 · miraheze/puppet · GitHub | github.com [04:48:50] [02miraheze/puppet] 07paladox 0328ea980 - Update mwtask141.yaml [04:48:51] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:48:51] [url] Page not found · GitHub · GitHub | github.com [04:49:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/28ea980c3709...4f4cd02bb9ca [04:49:00] [url] Comparing 28ea980c3709...4f4cd02bb9ca · miraheze/puppet · GitHub | github.com [04:49:01] [02miraheze/puppet] 07paladox 034f4cd02 - Update test131.yaml [04:49:03] [02puppet] 07paladox synchronize pull request 03#2768: mw121/22/31/32/41/44: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [04:49:03] [url] Page not found · GitHub · GitHub | github.com [04:50:23] !log depool and repool mw121 [04:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:50] !log depool and repool mw122 [04:52:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:53] !log depool and repool mw131 [04:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:41] !log depool and repool mw132 [04:54:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:54:39] !log depool and repool mw141 [04:55:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:55:24] !log depool and repool mw142 [04:55:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/72ae361b8724...f6a5040fd768 [04:57:17] [url] Comparing 72ae361b8724...f6a5040fd768 · miraheze/mw-config · GitHub | github.com [04:57:18] [02miraheze/mw-config] 07Universal-Omega 03f6a5040 - Update LocalSettings.php [04:58:28] miraheze/mw-config - Universal-Omega the build passed. [05:00:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:01:37] !log reboot mwtask141 [05:01:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:03] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.138 second response time [05:03:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [05:03:44] !log [@mwtask141] starting deploy of {'config': True} to all [05:03:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:25] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 41s [05:04:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:05:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.85, 2.15, 1.63 [05:07:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.75, 1.54 [05:09:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.54, 1.37, 1.43 [05:11:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:13:07] [02puppet] 07paladox edited pull request 03#2768: mw121/22/31/32/41/44|mwtask141|test131: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [05:13:07] [url] Page not found · GitHub · GitHub | github.com [05:13:12] [02puppet] 07paladox closed pull request 03#2768: mw121/22/31/32/41/44|mwtask141|test131: switch gluster volume to use load balanced version - 13https://github.com/miraheze/puppet/pull/2768 [05:13:12] [url] Page not found · GitHub · GitHub | github.com [05:13:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/miraheze/puppet/compare/7db3779e8434...e35420974de3 [05:13:14] [url] Comparing 7db3779e8434...e35420974de3 · miraheze/puppet · GitHub | github.com [05:13:15] [02miraheze/puppet] 07paladox 03e354209 - mw121/22/31/32/41/44|mwtask141|test131: switch gluster volume to use load balanced version (#2768) [05:13:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [05:13:18] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [05:13:19] [url] Page not found · GitHub · GitHub | github.com [05:15:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [05:17:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.30, 10.60, 11.85 [05:18:18] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw131 [05:19:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:23:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.98, 12.13, 11.89 [05:24:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.56, 8.19, 11.48 [05:24:20] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:24:29] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.50, 11.51, 11.72 [05:26:41] !log [@test131] starting deploy of {'config': True} to all [05:26:42] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:27:13] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:27:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.15, 12.14, 11.97 [05:27:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:05] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw132 mw142 [05:28:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:28:12] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [05:30:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.10, 11.92, 11.97 [05:32:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.43, 11.14, 11.69 [05:34:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.96, 12.97, 12.31 [05:36:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.43, 11.71, 11.97 [05:36:25] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:36:37] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:12] PROBLEM - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:47] PROBLEM - houkai2.cyou - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:07] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:19] PROBLEM - finalfantasy.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:35] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.340 second response time [05:38:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.01, 9.22, 11.27 [05:40:56] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:41:19] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.32, 11.41, 11.75 [05:42:11] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.022 second response time [05:42:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:43:13] PROBLEM - zh.internetpedia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:14] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:43:15] PROBLEM - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:24] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:43:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 10.70, 11.48 [05:44:34] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:32] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [05:48:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.66, 9.31, 10.16 [05:49:04] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:26] PROBLEM - polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:01] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:51:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.06, 8.58, 10.11 [05:52:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.80, 13.00, 11.49 [05:52:57] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:54:19] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 10 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [05:54:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.84, 11.00, 10.95 [05:54:32] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 10.37, 10.53 [05:56:28] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:56:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 18.88, 9.46, 4.04 [05:57:00] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:02] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:57:03] PROBLEM - wiki.cjgh.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:05] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw121 mw131 mw132 [05:57:15] PROBLEM - cp31 Stunnel HTTP for mwtask141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:21] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.64, 9.00, 9.99 [05:57:32] PROBLEM - mwtask141 php-fpm on mwtask141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:57:45] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:54] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [05:57:57] PROBLEM - mwtask141 NTP time on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:58] PROBLEM - cp30 Stunnel HTTP for mwtask141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.63, 13.29, 11.80 [05:58:52] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:59:04] PROBLEM - mwtask141 SSH on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:24] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:00:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.55, 11.63, 11.36 [06:00:50] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:39] PROBLEM - mwtask141 Check Gluster Clients on mwtask141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:02:45] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:02:47] PROBLEM - mwtask141 ferm_active on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:02:48] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:02:53] PROBLEM - mwtask141 nutcracker process on mwtask141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:03:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 21.91, 14.32, 11.71 [06:03:58] PROBLEM - mwtask141 conntrack_table_size on mwtask141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:04:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.77, 11.14, 11.14 [06:04:40] RECOVERY - mwtask141 NTP time on mwtask141 is OK: NTP OK: Offset -0.003766059875 secs [06:05:02] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [06:05:02] RECOVERY - mwtask141 ferm_active on mwtask141 is OK: OK ferm input default policy is set [06:05:02] RECOVERY - mwtask141 nutcracker process on mwtask141 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [06:05:16] RECOVERY - mwtask141 SSH on mwtask141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:05:49] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Thu 22 Sep 2022 04:19:34 GMT +0000. [06:05:53] RECOVERY - mwtask141 conntrack_table_size on mwtask141 is OK: OK: nf_conntrack is 0 % full [06:05:59] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [06:06:07] RECOVERY - mwtask141 php-fpm on mwtask141 is OK: PROCS OK: 3 processes with command name 'php-fpm7.4' [06:06:08] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 0.215 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::4c25,51.195.220.68 [06:06:08] RECOVERY - mwtask141 Check Gluster Clients on mwtask141 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [06:06:57] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [06:07:29] RECOVERY - houkai2.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'houkai2.cyou' will expire on Thu 27 Oct 2022 05:22:41 GMT +0000. [06:07:50] RECOVERY - cp31 Stunnel HTTP for mwtask141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.328 second response time [06:07:55] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.168 second response time [06:08:08] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.015 second response time [06:08:18] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.021 second response time [06:08:25] RECOVERY - cp30 Stunnel HTTP for mwtask141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.306 second response time [06:08:35] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:49] PROBLEM - vrcdev.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:52] PROBLEM - wiki.fossbots.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:11:36] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:11:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:11:39] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:22] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:23] PROBLEM - zh.internetpedia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zh.internetpedia.tk' expires in 13 day(s) (Mon 22 Aug 2022 22:07:13 GMT +0000). [06:13:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:14] RECOVERY - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Thu 01 Sep 2022 00:42:40 GMT +0000. [06:13:38] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 3.063 second response time [06:13:39] PROBLEM - phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:29] PROBLEM - grk.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:31] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:16:10] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:16:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.43, 9.79, 11.68 [06:16:41] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:22] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [06:18:25] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 11% [06:18:40] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.029 second response time [06:19:14] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:19:19] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 23.23, 14.29, 12.80 [06:21:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.71, 1.32 [06:21:27] PROBLEM - aresrocket.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:27] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:08] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.20, 11.14, 11.83 [06:22:26] PROBLEM - wiki.hsins.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:11] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.38, 1.23, 1.20 [06:23:36] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.777 second response time [06:24:16] PROBLEM - gp.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:49] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:57] PROBLEM - istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:31] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [06:25:54] PROBLEM - wiki.ravynos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.60, 12.09, 11.78 [06:26:22] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:23] RECOVERY - wiki.cjgh.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cjgh.xyz' will expire on Mon 03 Oct 2022 20:57:39 GMT +0000. [06:26:54] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:55] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:32] PROBLEM - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:28:35] PROBLEM - www.istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:14] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:30:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 10.77, 11.42 [06:30:38] PROBLEM - ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:45] PROBLEM - mcwiki.kirbygang.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:46] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:55] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:31:08] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:09] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:10] PROBLEM - slymods.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:27] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:32] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:52] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.042 second response time [06:34:01] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.23, 2.35, 3.97 [06:34:05] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [06:34:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 23.19, 14.12, 12.35 [06:34:30] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.035 second response time [06:34:34] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.014 second response time [06:35:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.78, 8.53, 11.30 [06:36:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:36:24] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:05] RECOVERY - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chouverse.cludisciples.cyou' will expire on Fri 04 Nov 2022 23:45:29 GMT +0000. [06:38:12] PROBLEM - wikifencing.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:17] RECOVERY - finalfantasy.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [06:38:17] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/0abf4d8b082a [06:38:19] [02miraheze/ssl] 07paladox 030abf4d8 - Disable events for betaheze.org [06:38:20] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [06:38:21] [url] Page not found · GitHub · GitHub | github.com [06:38:22] [02ssl] 07paladox opened pull request 03#585: Disable events for betaheze.org - 13https://github.com/miraheze/ssl/pull/585 [06:38:22] ... [06:38:26] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.022 second response time [06:39:13] PROBLEM - grayravens.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.91, 6.76, 9.93 [06:39:44] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6484a6129355...9afc89485f26 [06:39:45] [url] Comparing 6484a6129355...9afc89485f26 · miraheze/dns · GitHub | github.com [06:39:46] [02miraheze/dns] 07paladox 039afc894 - Add acme challenges to betaheze.org [06:39:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:40:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 10.52, 11.46 [06:41:08] [02ssl] 07paladox closed pull request 03#585: Disable events for betaheze.org - 13https://github.com/miraheze/ssl/pull/585 [06:41:08] PROBLEM - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:09] [url] Page not found · GitHub · GitHub | github.com [06:41:10] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [06:41:11] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ff19a2a1ce2d...ca0bd01ab581 [06:41:12] [url] Comparing ff19a2a1ce2d...ca0bd01ab581 · miraheze/ssl · GitHub | github.com [06:41:13] [02miraheze/ssl] 07paladox 03ca0bd01 - Disable events for betaheze.org (#585) [06:41:14] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [06:41:15] [url] Page not found · GitHub · GitHub | github.com [06:41:21] PROBLEM - metroidpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:21] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ca0bd01ab581...5aef2b1d3164 [06:41:21] [url] Comparing ca0bd01ab581...5aef2b1d3164 · miraheze/ssl · GitHub | github.com [06:41:22] [02miraheze/ssl] 07paladox 035aef2b1 - Update betaheze.org.crt [06:41:46] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:41:55] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:01] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.21, 2.51, 3.37 [06:42:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.66, 11.50, 11.63 [06:43:30] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [06:43:33] RECOVERY - grk.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.archiopedia.org' will expire on Sun 02 Oct 2022 05:34:49 GMT +0000. [06:43:35] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:43:36] PROBLEM - ecole.science - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:43:48] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.290 second response time [06:43:50] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:44:05] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:14] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [06:44:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 11.11, 11.53 [06:44:28] PROBLEM - phytinfo.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:52] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 1.70, 1.38 [06:45:42] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:05] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:31] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:33] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:47] PROBLEM - wiki.aridia.space - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:58] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.50, 1.34 [06:47:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 27.16, 16.49, 12.44 [06:47:42] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:55] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Sun 06 Nov 2022 03:48:13 GMT +0000. [06:48:02] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.033 second response time [06:48:28] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:02] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:20] PROBLEM - pwsc.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:20] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.045 second response time [06:50:42] PROBLEM - wiki.funkey-project.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:44] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:01] RECOVERY - aresrocket.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'aresrocket.ml' will expire on Fri 14 Oct 2022 05:26:01 GMT +0000. [06:51:25] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Sun 06 Nov 2022 01:58:31 GMT +0000. [06:51:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.87, 10.78, 11.09 [06:51:28] RECOVERY - wiki.hsins.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hsins.eu' will expire on Wed 05 Oct 2022 21:47:47 GMT +0000. [06:52:00] PROBLEM - wiki.thehall.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:05] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ipv6bolivia.tk' expires in 14 day(s) (Tue 23 Aug 2022 13:09:22 GMT +0000). [06:52:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.00, 7.86, 9.73 [06:55:02] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Sat 05 Nov 2022 07:04:32 GMT +0000. [06:55:07] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.022 second response time [06:55:19] RECOVERY - istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [06:55:26] RECOVERY - wiki.ravynos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ravynos.com' will expire on Thu 01 Sep 2022 00:45:42 GMT +0000. [06:55:28] RECOVERY - betaheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'betaheze.org' will expire on Mon 07 Nov 2022 05:40:44 GMT +0000. [06:55:36] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Sat 01 Oct 2022 16:25:42 GMT +0000. [06:55:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.39, 1.35, 1.02 [06:56:21] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Sun 06 Nov 2022 01:58:55 GMT +0000. [06:56:29] RECOVERY - threedomwiki.pcast.site - LetsEncrypt on sslhost is OK: OK - Certificate 'threedomwiki.pcast.site' will expire on Fri 04 Nov 2022 21:32:05 GMT +0000. [06:56:41] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [06:56:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.38, 8.43, 9.94 [06:56:48] PROBLEM - intp.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:56:56] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:00] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:39] PROBLEM - hi.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:44] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:48] RECOVERY - www.istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [06:59:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.94, 9.19, 9.39 [06:59:26] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [06:59:39] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:44] RECOVERY - ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Fri 04 Nov 2022 21:29:15 GMT +0000. RECOVERY - slymods.info - LetsEncrypt on sslhost is OK: OK - Certificate 'slymods.info' will expire on Fri 21 Oct 2022 06:31:32 GMT +0000. [07:00:00] RECOVERY - mcwiki.kirbygang.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'mcwiki.kirbygang.tk' will expire on Wed 05 Oct 2022 22:00:10 GMT +0000. [07:00:15] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sun 02 Oct 2022 10:22:40 GMT +0000. [07:01:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.00, 7.44, 8.72 [07:01:34] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [07:03:01] PROBLEM - www.proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:37] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:43] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: CRITICAL - load average: 11.68, 4.94, 2.00 [07:03:43] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:44] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:04:59] PROBLEM - prometheus131 NTP time on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:39] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:44] RECOVERY - vrcdev.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'vrcdev.wiki' will expire on Tue 20 Sep 2022 12:25:22 GMT +0000. [07:06:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.65, 2.19, 3.77 [07:07:39] PROBLEM - prometheus131 PowerDNS Recursor on prometheus131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:07:55] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.065 second response time [07:08:07] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Sat 05 Nov 2022 07:41:49 GMT +0000. [07:08:16] RECOVERY - wikifencing.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wikifencing.ru' will expire on Tue 06 Sep 2022 20:55:14 GMT +0000. [07:08:37] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.026 second response time [07:08:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.99, 4.89, 4.60 [07:09:09] PROBLEM - prometheus131 ferm_active on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:15] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:15] PROBLEM - prometheus131 conntrack_table_size on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:45] RECOVERY - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssangyongsports.eu.org' will expire on Wed 31 Aug 2022 22:12:11 GMT +0000. [07:09:49] RECOVERY - wiki.fossbots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fossbots.org' will expire on Sun 16 Oct 2022 13:11:42 GMT +0000. [07:10:17] RECOVERY - metroidpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'metroidpedia.com' will expire on Wed 26 Oct 2022 18:29:18 GMT +0000. [07:10:32] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [07:10:39] RECOVERY - burnout.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'burnout.wiki' will expire on Tue 01 Nov 2022 04:10:29 GMT +0000. [07:12:20] RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Fri 04 Nov 2022 20:42:01 GMT +0000. [07:13:07] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:16] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [07:13:39] RECOVERY - prometheus131 NTP time on prometheus131 is OK: NTP OK: Offset -0.003255605698 secs [07:13:39] RECOVERY - phytinfo.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'phytinfo.cf' will expire on Thu 27 Oct 2022 15:20:29 GMT +0000. [07:13:47] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:13:48] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:15:07] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.022 second response time [07:15:15] RECOVERY - prometheus131 ferm_active on prometheus131 is OK: OK ferm input default policy is set [07:15:15] RECOVERY - prometheus131 conntrack_table_size on prometheus131 is OK: OK: nf_conntrack is 0 % full [07:15:15] RECOVERY - prometheus131 PowerDNS Recursor on prometheus131 is OK: DNS OK: 5.783 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:15:15] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [07:15:22] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Thu 13 Oct 2022 04:22:47 GMT +0000. [07:15:22] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Mon 10 Oct 2022 04:52:56 GMT +0000. [07:16:02] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:16:15] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Fri 04 Nov 2022 23:48:53 GMT +0000. [07:16:33] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:16:34] RECOVERY - wiki.aridia.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Sat 24 Sep 2022 13:17:20 GMT +0000. [07:16:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.01, 3.28, 3.94 [07:16:56] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:05] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Fri 04 Nov 2022 22:04:13 GMT +0000. [07:17:20] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Fri 30 Sep 2022 17:38:06 GMT +0000. [07:17:29] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:57] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [07:18:32] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:18:52] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:18:53] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.283 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [07:19:03] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.054 second response time [07:19:09] RECOVERY - pwsc.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwsc.polandballwiki.com' will expire on Sun 02 Oct 2022 10:41:20 GMT +0000. [07:19:40] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Fri 04 Nov 2022 22:27:36 GMT +0000. [07:19:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.86, 1.52, 1.15 [07:20:33] RECOVERY - wiki.funkey-project.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Sat 05 Nov 2022 07:45:05 GMT +0000. [07:20:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.56, 3.86, 3.99 [07:20:53] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:21:32] RECOVERY - wiki.thehall.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thehall.xyz' will expire on Sat 05 Nov 2022 09:16:24 GMT +0000. [07:21:37] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.021 second response time [07:21:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.26, 1.09 [07:21:46] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:08] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:10] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:29:43 GMT +0000. [07:22:32] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.039 second response time [07:23:11] PROBLEM - tech.teojingyao.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.83, 0.49, 0.54 [07:24:55] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.059 second response time [07:25:33] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:25:45] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:25:53] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/867f96af9c91 [07:25:54] [02miraheze/dns] 07paladox 03867f96a - Depool cp20/21 [07:25:56] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [07:25:56] [url] Page not found · GitHub · GitHub | github.com [07:25:57] [02dns] 07paladox opened pull request 03#328: Depool cp20/21 - 13https://github.com/miraheze/dns/pull/328 [07:25:58] [url] Page not found · GitHub · GitHub | github.com [07:26:01] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [07:26:46] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [07:26:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 9.52, 8.11 [07:26:57] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:00] PROBLEM - en.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:08] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:16] PROBLEM - wiki.simorgh.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:30] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [07:27:40] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 12 Oct 2022 13:21:03 GMT +0000. [07:27:44] [02dns] 07paladox edited pull request 03#328: Depool cp20/21 - 13https://github.com/miraheze/dns/pull/328 [07:27:45] [url] Page not found · GitHub · GitHub | github.com [07:28:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:28:05] [02dns] 07paladox closed pull request 03#328: Depool cp20/21 - 13https://github.com/miraheze/dns/pull/328 [07:28:05] ... [07:28:06] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9afc89485f26...bd7aba33f5da [07:28:07] [url] Comparing 9afc89485f26...bd7aba33f5da · miraheze/dns · GitHub | github.com [07:28:08] [02miraheze/dns] 07paladox 03bd7aba3 - Depool cp20/21 (#328) [07:28:09] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [07:28:11] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [07:28:11] [url] Page not found · GitHub · GitHub | github.com [07:28:27] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:46] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:28:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.65, 12.03, 9.27 [07:29:04] PROBLEM - www.istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:30:50] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 4 % full [07:30:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.20, 9.59, 8.70 [07:31:09] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:31:50] PROBLEM - dcmultiverse.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:20] RECOVERY - www.proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [07:32:27] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:42] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:34:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.020 second response time [07:35:55] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.053 second response time [07:36:52] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.49, 1.16 [07:37:12] PROBLEM - prometheus131 Current Load on prometheus131 is WARNING: WARNING - load average: 0.10, 0.86, 3.87 [07:37:42] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:37:50] RECOVERY - grayravens.com - LetsEncrypt on sslhost is OK: OK - Certificate 'grayravens.com' will expire on Sun 18 Sep 2022 13:03:26 GMT +0000. [07:38:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 1.67, 1.26 [07:38:55] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [07:39:06] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:39:29] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:32] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:35] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.47, 1.24 [07:41:10] RECOVERY - prometheus131 Current Load on prometheus131 is OK: OK - load average: 0.23, 0.49, 3.02 [07:41:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.017 second response time [07:41:30] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:57] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.288 second response time [07:41:59] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Sun 06 Nov 2022 02:11:45 GMT +0000. [07:42:34] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.033 second response time [07:43:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.81, 1.42 [07:43:31] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.046 second response time [07:43:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:43:39] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:43:45] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:45:44] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [07:45:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.83, 1.82, 1.57 [07:47:11] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [07:47:42] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.023 second response time [07:48:17] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:42] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.023 second response time [07:48:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.64, 1.53 [07:49:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.04, 2.91, 3.80 [07:50:12] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.050 second response time [07:50:17] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [07:51:09] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ipv6bolivia.tk' expires in 14 day(s) (Tue 23 Aug 2022 13:09:22 GMT +0000). [07:51:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.32, 3.97 [07:51:42] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 10 day(s) (Fri 19 Aug 2022 17:20:05 GMT +0000). [07:51:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.76, 4.01 [07:52:27] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:52:39] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.250 second response time [07:52:45] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.91, 1.68 [07:52:46] RECOVERY - tech.teojingyao.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tech.teojingyao.com' will expire on Sat 08 Oct 2022 04:57:57 GMT +0000. [07:52:48] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [07:52:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:54:28] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.015 second response time [07:54:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.61, 1.59 [07:54:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:54:54] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 44% [07:55:33] RECOVERY - en.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiretet.ga' will expire on Sun 02 Oct 2022 05:37:01 GMT +0000. [07:55:38] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Sat 05 Nov 2022 08:55:30 GMT +0000. [07:55:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.59, 3.97 [07:55:45] RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Wed 28 Sep 2022 11:34:56 GMT +0000. [07:56:29] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Sat 05 Nov 2022 07:04:32 GMT +0000. [07:56:55] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 12% [07:56:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.90, 1.58 [07:57:06] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:09] RECOVERY - hi.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.famepedia.org' will expire on Tue 18 Oct 2022 14:04:28 GMT +0000. [07:57:10] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Sun 02 Oct 2022 10:55:38 GMT +0000. [07:57:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 3.18, 3.62 [07:57:46] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [07:57:52] RECOVERY - www.istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [07:58:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [07:59:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.25, 3.60 [08:00:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.67, 1.57 [08:00:47] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:01:09] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: CRITICAL - load average: 4.02, 2.35, 1.91 [08:01:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 2.79, 3.39 [08:01:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.317 second response time [08:03:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.58, 3.78, 3.71 [08:04:18] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:05:15] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:06] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:08] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:08:21] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:17] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.025 second response time [08:09:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.55, 3.88, 3.89 [08:09:45] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.060 second response time [08:09:56] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.320 second response time [08:10:15] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.245 second response time [08:10:21] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:24] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:11:24] RECOVERY - prometheus131 Current Load on prometheus131 is OK: OK - load average: 1.15, 3.34, 3.03 [08:12:20] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.037 second response time [08:13:14] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [08:13:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 4.47, 4.08 [08:13:21] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:14:43] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:15:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.83, 3.89 [08:15:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.16, 5.25, 4.06 [08:16:17] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:40] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.030 second response time [08:17:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.76, 3.95, 3.89 [08:17:19] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:11] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [08:18:22] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [08:18:58] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:19:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.26, 3.22, 3.63 [08:19:16] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:19:45] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:20:37] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.258 second response time [08:21:02] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.062 second response time [08:21:13] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.99, 1.64, 1.52 [08:21:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.26, 2.63, 3.37 [08:21:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.37, 2.10, 1.75 [08:21:31] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:03] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 5.092 second response time [08:22:20] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:23:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.48, 1.48 [08:23:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.91, 3.87 [08:24:22] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [08:25:14] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:27:12] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.024 second response time [08:27:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.28, 2.69, 3.40 [08:28:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:28:46] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.030 second response time [08:29:39] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:37] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 14% [08:32:06] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:56] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:38] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.55, 10.63, 8.10 [08:35:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 22.42, 10.26, 5.65 [08:35:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.14, 6.31, 4.53 [08:35:57] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:24] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [08:36:38] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:38:26] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 2.057 second response time [08:38:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.27, 10.73, 8.79 [08:39:39] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:39:43] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.027 second response time [08:40:37] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.00, 10.12, 8.83 [08:40:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:41:15] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:41:47] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:41:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.51, 12.22, 9.22 [08:42:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [08:42:08] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8828 MB (22% inode=96%); [08:42:39] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.064 second response time [08:43:20] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 7.073 second response time [08:43:31] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.120 second response time [08:43:35] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.25, 11.60, 9.40 [08:44:02] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:44:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:44:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [08:44:57] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:33] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9252 MB (24% inode=96%); [08:45:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.03, 0.02, 0.00 [08:46:14] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.025 second response time [08:46:16] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:46:39] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [08:46:52] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [08:47:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.66, 2.07, 1.62 [08:47:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.58, 9.89, 9.22 [08:48:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:48:47] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [08:48:58] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:50:12] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:51:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [08:52:08] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.029 second response time [08:53:32] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.24, 9.93, 9.31 [08:54:00] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:03] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw132 mw142 [08:54:24] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 14% [08:55:14] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 1.050 second response time [08:55:28] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [08:55:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.08, 11.29, 9.91 [08:55:59] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.019 second response time [08:56:02] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [08:56:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.47, 12.30, 10.47 [08:56:35] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:41] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.57, 11.04, 10.00 [08:58:05] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 24% [08:58:17] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 11.46, 10.38 [08:58:38] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:01:22] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.31, 9.87, 9.76 [09:02:09] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:02:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.46, 12.24, 10.95 [09:03:20] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:29] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:04:23] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:50] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:03] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [09:06:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 10.90, 10.74 [09:06:23] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.020 second response time [09:07:18] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:48] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:08:19] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.060 second response time [09:08:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.01, 0.02, 0.02 [09:08:57] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.022 second response time [09:09:14] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.142 second response time [09:09:20] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.772 second response time [09:09:41] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.049 second response time [09:09:43] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.131 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:09:43] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [09:09:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.52, 10.99, 10.18 [09:11:36] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.020 second response time [09:11:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.91, 9.43, 10.20 [09:11:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.60, 9.85, 9.87 [09:12:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.83, 1.42, 1.88 [09:12:56] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:53] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 57% [09:14:12] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.058 second response time [09:14:55] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:15:48] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:11] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:16:50] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:03] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:15] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:18:05] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:18:24] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.81, 1.84, 1.88 [09:18:40] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 17% [09:18:54] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 16% [09:19:17] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:21] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.45, 10.57, 10.30 [09:19:37] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:53] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:20:39] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [09:21:15] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [09:21:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 10.00, 10.11 [09:21:49] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8811 MB (22% inode=96%); [09:21:58] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:22:04] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.045 second response time [09:23:53] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.050 second response time [09:24:05] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:24:25] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 10.42, 9.67 [09:24:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.13, 10.26, 10.21 [09:25:23] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:26:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:26:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.68, 8.97, 9.25 [09:27:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.60, 8.91, 9.72 [09:27:15] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9241 MB (24% inode=96%); [09:27:19] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.208 second response time [09:27:50] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.033 second response time [09:27:51] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:28:03] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 44% [09:28:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.93, 1.64, 1.87 [09:28:56] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [09:30:17] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 29% [09:30:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.85, 1.91 [09:30:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.68, 1.95 [09:30:53] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:31:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 1.047 second response time [09:31:15] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.092 second response time [09:32:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.74, 1.85 [09:32:53] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 16% [09:33:03] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.085 second response time [09:33:40] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:34:44] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:35:11] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:36:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [09:36:07] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.032 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:36:13] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:36:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.45, 1.83, 1.91 [09:37:42] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:58] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:38:14] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:28] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:38:32] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:38:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:52] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:19] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.047 second response time [09:39:29] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:39:44] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [09:40:08] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [09:40:10] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.074 second response time [09:40:14] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:30] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.061 second response time [09:40:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.52, 1.68 [09:40:31] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:40:35] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.026 second response time [09:40:47] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:52] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.147 second response time [09:41:13] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [09:42:10] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.431 second response time [09:42:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.42, 1.83, 1.93 [09:42:46] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [09:43:01] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 43% [09:43:06] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:44:08] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:44:14] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.066 second response time [09:44:16] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.050 second response time [09:44:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.65, 1.72 [09:45:07] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.079 second response time [09:45:31] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 24% [09:45:48] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.068 second response time [09:45:51] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:11] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.086 second response time [09:46:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.45, 1.63 [09:47:25] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:04] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 16% [09:48:57] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:50:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.91, 1.78 [09:50:32] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.31, 2.13, 1.92 [09:51:11] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.043 second response time [09:51:14] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:51:41] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.252 second response time [09:52:07] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.86, 1.78 [09:52:33] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.228 second response time [09:54:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.98, 1.91 [09:54:54] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:54:55] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:27] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:19] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.51, 1.66 [09:56:53] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 33% [09:57:42] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:16] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:58:23] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.037 second response time [09:58:30] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [09:59:01] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:59:59] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.027 second response time [10:00:26] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.82, 1.76 [10:00:37] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.049 second response time [10:01:47] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.59, 1.40, 1.61 [10:02:32] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.123 second response time [10:02:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.91, 1.37, 1.63 [10:03:10] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:03:12] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:03:51] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:03:51] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.059 second response time [10:03:54] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [10:05:10] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [10:05:18] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:24] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [10:05:25] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.051 second response time [10:05:51] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:58] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:03] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001270800829 secs [10:06:03] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.55, 1.89, 1.75 [10:06:38] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:47] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:38] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:48] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:07:56] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 24% [10:08:12] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [10:08:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.80, 1.73 [10:08:40] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.168 second response time [10:08:55] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:00] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:26] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:34] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.029 second response time [10:09:37] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.090 second response time [10:10:15] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.022 second response time [10:10:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.50, 1.67, 1.69 [10:11:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.037 second response time [10:12:17] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:13:48] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.034 second response time [10:14:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:14:16] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.017 second response time [10:14:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.95, 1.78 [10:14:55] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:24] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.189 second response time [10:15:42] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:50] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:03] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:19] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:30] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [10:17:38] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [10:17:39] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.037 second response time [10:17:47] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [10:18:18] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:18:48] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:18:52] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:20:05] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:20:48] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [10:21:19] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [10:21:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:11] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:22:53] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.031 second response time [10:23:57] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:24:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.027 second response time [10:26:09] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.048 second response time [10:26:24] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.83, 1.92 [10:26:33] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:43] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:50] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8799 MB (22% inode=96%); [10:26:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [10:28:02] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.97, 2.65, 2.18 [10:28:45] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.025 second response time [10:28:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:30:06] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [10:30:26] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:30:28] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.015 second response time [10:30:28] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:30:29] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.38, 2.01, 1.70 [10:31:58] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:32:43] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:33] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:41] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [10:33:54] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [10:34:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.85, 1.95 [10:34:41] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.063 second response time [10:34:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.48, 1.80, 1.69 [10:34:51] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:20] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.50, 0.25, 0.10 [10:35:46] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39163 bytes in 0.035 second response time [10:35:50] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:36:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.016 second response time [10:36:51] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:36:56] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.085 second response time [10:38:38] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:38:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.57, 1.63 [10:40:07] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:17] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:40:29] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.95, 1.95 [10:40:35] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 48% [10:40:47] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:49] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [10:40:56] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:41:04] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:27] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.015 second response time [10:42:41] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.039 second response time [10:42:44] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:44] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:43:04] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 23% [10:43:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:50] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.019 second response time [10:43:59] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.22, 2.01, 1.79 [10:45:03] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:45:12] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [10:45:17] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.053 second response time [10:45:31] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.112 second response time [10:45:57] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [10:45:57] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:46:33] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [10:46:57] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:03] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.100 second response time [10:48:22] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:48:24] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.040 second response time [10:48:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.88, 1.93 [10:48:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.93, 1.86, 1.77 [10:49:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.22, 3.88 [10:49:25] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [10:49:58] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.91, 1.93 [10:50:40] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:43] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.178 second response time [10:50:54] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 2.071 second response time [10:50:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:51:12] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:52:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.47, 1.63 [10:52:51] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:52:56] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:53:23] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8793 MB (22% inode=96%); [10:54:49] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.038 second response time [10:54:51] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.035 second response time [10:55:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:09] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [10:55:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 4.46, 4.19 [10:55:34] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0006691813469 secs [10:56:06] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.70, 3.95 [10:58:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.94, 1.98 [10:58:43] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [10:59:13] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.015 second response time [10:59:35] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:59:46] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:36] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:54] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:55] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:56] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.020 second response time [11:01:10] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:01:15] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:01:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.54, 5.38, 4.46 [11:02:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:38] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [11:02:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.029 second response time [11:03:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.03, 0.05 [11:03:13] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.053 second response time [11:03:24] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:56] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:04:42] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.058 second response time [11:05:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:05:23] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [11:05:46] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [11:05:57] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.215 second response time [11:06:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:06:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.93, 1.91 [11:07:05] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.096 second response time [11:07:10] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:11] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:08:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.63, 1.80 [11:08:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 11.01, 9.12 [11:09:06] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.023 second response time [11:09:10] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:10:02] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:10:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.34, 9.96, 8.96 [11:11:10] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [11:11:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:12:02] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.037 second response time [11:12:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:12:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.74, 1.80 [11:13:05] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [11:13:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [11:14:25] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.03, 0.00 [11:14:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.20, 1.60, 1.74 [11:14:45] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:15:28] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:48] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0007024407387 secs [11:17:24] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.046 second response time [11:18:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:18:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.94, 1.86 [11:19:53] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:48] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:49] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:22:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 1.72, 1.47 [11:22:47] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:50] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.063 second response time [11:23:12] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:30] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [11:24:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.53, 1.43 [11:25:01] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.070 second response time [11:25:13] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.044 second response time [11:26:50] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.79, 2.00 [11:28:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [11:29:04] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:44] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:52] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.93, 1.90, 2.01 [11:30:53] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:34] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 20% [11:31:52] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:31:55] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.00151103735 secs [11:31:56] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.031 second response time [11:32:56] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.048 second response time [11:33:49] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.034 second response time [11:34:04] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.04, 10.48, 9.66 [11:35:34] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [11:36:07] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:36:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.89, 1.99 [11:37:16] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [11:38:13] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:38:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.06, 10.18, 9.76 [11:39:07] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:15] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 17% [11:39:26] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:30] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.302 second response time [11:40:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 1.86, 1.93 [11:41:12] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:41:41] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:42] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:13] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.95, 11.22, 10.27 [11:43:48] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [11:43:57] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.544 second response time [11:44:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:23] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.054 second response time [11:44:50] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:44:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.36, 9.78, 9.89 [11:45:19] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.83, 1.65 [11:45:55] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.338 second response time [11:46:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.024 second response time [11:46:17] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [11:46:33] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:47:17] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.041 second response time [11:47:58] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:48:07] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:30] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.547 second response time [11:49:01] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.204 second response time [11:49:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.24, 2.39, 1.89 [11:50:04] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:13] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.326 second response time [11:51:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.55, 11.45, 9.76 [11:51:59] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [11:52:34] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:17] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:01] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:17] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:55:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.73, 11.16, 10.04 [11:55:19] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.017 second response time [11:55:34] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:35] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:20] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:44] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.509 second response time [11:56:44] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:57:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.90, 13.14, 10.92 [11:57:48] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 32% [11:58:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:58:41] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [11:58:42] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:13] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.038 second response time [11:59:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.841 second response time [12:00:06] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:05] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:13] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:22] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.366 second response time [12:01:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 11.53, 10.55 [12:01:31] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 2.076 second response time [12:01:35] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [12:02:00] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.133 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:02:03] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.22, 11.65, 11.07 [12:03:18] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.062 second response time [12:03:34] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.174 second response time [12:03:36] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [12:03:53] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:32] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 11.66, 11.12 [12:05:01] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:12] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.250 second response time [12:06:30] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.047 second response time [12:06:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 11.13, 10.97 [12:07:03] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.159 second response time [12:07:16] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.74, 9.94, 10.19 [12:07:22] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:16] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.044 second response time [12:08:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.91, 12.08, 11.38 [12:08:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.72, 1.93 [12:08:45] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.046 second response time [12:09:05] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:09:31] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:09:33] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.089 second response time [12:09:40] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:02] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:10:50] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.48, 12.06, 11.00 [12:12:00] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.047 second response time [12:12:45] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.019 second response time [12:13:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 10.93, 10.70 [12:13:50] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.058 second response time [12:14:09] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:14:24] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:14:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.44, 1.86, 1.89 [12:14:54] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:16:26] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.708 second response time [12:16:35] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [12:16:37] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:43] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:16:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.27, 11.47, 10.95 [12:18:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.00, 11.33, 10.97 [12:19:54] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.13, 12.41, 11.42 [12:21:20] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.182 second response time [12:21:38] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:39] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8778 MB (22% inode=96%); [12:21:39] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:49] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [12:21:53] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [12:22:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:16] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:23:38] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [12:23:48] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.079 second response time [12:23:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.33, 11.28, 11.94 [12:24:12] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 49% [12:24:37] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:25:21] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.276 second response time [12:25:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.86, 11.91, 12.11 [12:26:31] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.062 second response time [12:26:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.95, 11.96, 11.71 [12:27:10] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:28] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:30] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:31] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:42] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:57] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [12:29:34] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:40] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 22% [12:30:33] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [12:30:36] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:38] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 1.553 second response time [12:30:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:51] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.98, 12.29, 11.94 [12:30:59] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8776 MB (22% inode=96%); [12:31:07] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:31] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:13] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:32] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.912 second response time [12:32:34] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 2.705 second response time [12:32:45] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.193 second response time [12:32:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:48] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.354 second response time [12:33:03] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:33:22] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.050 second response time [12:33:24] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:33:25] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:33:26] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.356 second response time [12:33:30] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.009 second response time [12:33:58] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [12:34:09] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.036 second response time [12:34:42] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.027 second response time [12:35:22] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.056 second response time [12:35:35] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.155 second response time [12:35:39] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.016 second response time [12:35:41] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.073 second response time [12:35:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 10.97, 11.79 [12:38:19] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:39:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.64, 13.51, 12.50 [12:40:06] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [12:40:32] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:09] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:19] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:27] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:33] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:17] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:25] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [12:44:38] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:45:30] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.083 second response time [12:45:51] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 2.037 second response time [12:45:52] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [12:46:14] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 32% [12:46:40] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.062 second response time [12:49:13] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.046 second response time [12:50:19] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:26] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.091 second response time [12:51:06] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:52:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:52:33] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 32% [12:54:27] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:54:40] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.032 second response time [12:55:07] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:12] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.016 second response time [12:55:37] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:57] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.059 second response time [12:56:21] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:25] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.693 second response time [12:56:26] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:32] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [12:58:22] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [12:58:26] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:55] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.053 second response time [12:59:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.128 second response time [13:00:37] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [13:00:43] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:32] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:51] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:01] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:34] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:03:36] PROBLEM - prometheus131 PowerDNS Recursor on prometheus131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:04:06] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: CRITICAL - load average: 11.62, 5.57, 2.32 [13:04:12] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.051 second response time [13:04:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.03, 0.06, 0.03 [13:04:47] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8769 MB (22% inode=96%); [13:04:56] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:01] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:35] RECOVERY - prometheus131 PowerDNS Recursor on prometheus131 is OK: DNS OK: 0.165 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:06:12] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.025 second response time [13:07:09] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:07:10] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [13:07:11] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:07:44] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:07:53] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:08:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:00] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.039 second response time [13:09:10] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.020 second response time [13:09:35] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [13:10:13] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.034 second response time [13:10:23] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.242 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:10:48] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:48] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [13:13:06] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.015 second response time [13:14:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [13:15:15] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:39] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [13:17:17] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.284 second response time [13:17:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 46% [13:18:11] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:19:31] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:05] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [13:20:40] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 28% [13:20:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 10.18, 11.67 [13:21:27] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:21:42] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:21:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.92, 9.88, 11.87 [13:22:40] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:37] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [13:23:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.52, 10.95, 12.02 [13:24:35] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:24:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 10.10, 11.23 [13:25:02] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.73, 10.65, 11.77 [13:26:42] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:27:02] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.904 second response time [13:27:43] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:49] PROBLEM - prometheus131 PowerDNS Recursor on prometheus131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:27:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.56, 12.10, 12.21 [13:28:05] PROBLEM - prometheus131 NTP time on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:54] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 10.91, 11.39 [13:29:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.47, 11.41, 11.97 [13:30:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:12] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:12] PROBLEM - prometheus131 conntrack_table_size on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:14] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [13:31:34] PROBLEM - prometheus131 ferm_active on prometheus131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:43] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [13:31:46] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [13:31:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.32, 11.82, 12.00 [13:31:57] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:48] RECOVERY - prometheus131 conntrack_table_size on prometheus131 is OK: OK: nf_conntrack is 0 % full [13:32:49] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [13:32:52] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:32:56] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:33:52] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [13:34:00] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.036 second response time [13:34:18] RECOVERY - prometheus131 ferm_active on prometheus131 is OK: OK ferm input default policy is set [13:34:18] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 30% [13:34:19] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.224 second response time [13:34:22] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:24] RECOVERY - prometheus131 PowerDNS Recursor on prometheus131 is OK: DNS OK: 0.135 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:34:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.59, 1.66, 1.99 [13:34:50] RECOVERY - prometheus131 NTP time on prometheus131 is OK: NTP OK: Offset -0.0009737610817 secs [13:34:55] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 11.32, 11.73 [13:36:51] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:05] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.15, 1.84, 1.98 [13:38:54] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.96, 8.26, 10.03 [13:39:04] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.88, 12.55, 12.04 [13:39:59] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [13:40:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:40:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.02, 1.50, 1.84 [13:40:50] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:02] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.053 second response time [13:41:07] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.046 second response time [13:41:57] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.042 second response time [13:42:02] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:42:13] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:42:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [13:42:37] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.038 second response time [13:42:42] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:42:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.86, 11.08, 10.74 [13:42:56] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:02] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [13:43:29] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 42% [13:43:58] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.138 second response time [13:44:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:44:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:44:19] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 42% [13:44:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 11.68, 11.04 [13:45:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [13:46:15] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.016 second response time [13:46:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:46:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.45, 1.42, 1.68 [13:47:22] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:25] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:55] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:09] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:20] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.648 second response time [13:48:32] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 23% [13:48:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.48, 12.27, 11.36 [13:49:12] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.083 second response time [13:49:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:49:51] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 18% [13:49:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.82, 11.38, 11.90 [13:49:55] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.017 second response time [13:50:02] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:05] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.422 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:50:25] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.074 second response time [13:50:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.28, 1.81, 1.77 [13:50:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 11.91, 11.34 [13:51:20] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:51:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.18, 11.93, 12.04 [13:52:30] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.014 second response time [13:52:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.99, 1.82, 1.78 [13:52:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.51, 12.62, 11.65 [13:53:16] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:54:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.23, 1.89, 1.81 [13:55:58] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:56:05] PROBLEM - prometheus131 PowerDNS Recursor on prometheus131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:56:16] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.057 second response time [13:56:25] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:57:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:50] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:00] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:12] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:19] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.28, 10.37, 11.15 [13:59:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:59:47] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:59:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.56, 10.63, 11.71 [14:00:11] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.028 second response time [14:00:17] RECOVERY - prometheus131 PowerDNS Recursor on prometheus131 is OK: DNS OK: 0.313 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:00:24] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.548 second response time [14:00:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.028 second response time [14:01:09] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:15] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.040 second response time [14:01:20] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.020 second response time [14:01:35] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:19] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:02:36] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:48] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:09] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:31] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:40] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.498 second response time [14:03:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:12] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:37] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.02, 10.28, 10.74 [14:05:02] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 43% [14:05:27] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.194 second response time [14:05:44] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [14:05:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.64, 11.46, 11.64 [14:06:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [14:07:04] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.229 second response time [14:07:13] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:07:33] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:07:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.40, 11.63, 11.69 [14:08:15] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:08:16] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.038 second response time [14:08:17] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:50] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.702 second response time [14:09:01] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.992 second response time [14:09:33] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.795 second response time [14:09:33] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.059 second response time [14:09:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.21, 12.75, 12.08 [14:10:26] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:30] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:52] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.156 second response time [14:10:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.08, 9.93, 10.47 [14:11:04] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:11:55] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [14:11:56] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:29] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.015 second response time [14:12:45] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.063 second response time [14:12:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.77, 11.34, 10.94 [14:13:30] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 3.088 second response time [14:13:55] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.143 second response time [14:14:12] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [14:14:46] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.77, 10.11, 10.55 [14:15:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 58% [14:15:46] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 50% [14:16:06] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:16:39] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 11.25, 10.93 [14:17:08] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:49] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:41] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.043 second response time [14:18:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.94, 10.03, 10.53 [14:19:39] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [14:19:55] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.009 second response time [14:20:20] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:43] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [14:21:08] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.018 second response time [14:21:16] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:28] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:22:28] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [14:22:33] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.030 second response time [14:22:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.78, 9.29, 10.11 [14:23:08] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:14] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.023 second response time [14:23:15] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:23:26] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9182 MB (23% inode=96%); [14:24:47] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:24:57] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:04] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.327 second response time [14:25:07] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:25:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:25:34] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:14] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 10.15, 10.33 [14:27:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.18, 0.16, 0.06 [14:27:23] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:27:38] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8753 MB (22% inode=96%); [14:28:35] PROBLEM - prometheus131 PowerDNS Recursor on prometheus131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:28:36] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.01, 13.06, 11.40 [14:29:07] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:30] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [14:29:40] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.050 second response time [14:30:00] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.015 second response time [14:30:40] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:30:40] RECOVERY - prometheus131 PowerDNS Recursor on prometheus131 is OK: DNS OK: 2.738 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:30:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:30:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.37, 11.49, 11.03 [14:31:00] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:28] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:33] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 34% [14:32:23] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:32:29] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.76, 13.37, 11.78 [14:33:13] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.058 second response time [14:33:27] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:42] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:34:31] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [14:35:19] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.071 second response time [14:35:27] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.158 second response time [14:35:34] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:35:59] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:36:19] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:22] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.070 second response time [14:36:32] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:37:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3388 bytes in 9.137 second response time [14:37:39] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [14:37:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 11.16, 11.95 [14:37:59] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.175 second response time [14:38:27] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:38:31] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.082 second response time [14:38:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 11.16, 11.41 [14:39:02] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.210 second response time [14:39:45] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:49] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 33% [14:40:50] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [14:40:56] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.036 second response time [14:41:03] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:49] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [14:41:52] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.62, 12.58, 12.27 [14:42:00] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:42:04] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:44:06] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.152 second response time [14:44:09] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:13] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [14:44:17] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.019 second response time [14:44:27] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:03] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:46:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:46:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.18, 8.34, 10.07 [14:47:09] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:47:16] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.073 second response time [14:47:48] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:48:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [14:48:34] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:02] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:13] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:50:41] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [14:50:42] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:51:17] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 29% [14:51:36] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:51:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:05] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [14:52:42] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.068 second response time [14:52:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:52:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.73, 10.93, 10.65 [14:53:00] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 3.043 second response time [14:53:53] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.015 second response time [14:54:25] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.02, 0.04, 0.00 [14:55:54] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.135 second response time [14:56:37] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.019 second response time [14:56:41] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.28, 10.39, 10.61 [14:58:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3388 bytes in 4.878 second response time [14:58:51] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.091 second response time [14:58:52] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:41] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:01:27] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:32] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [15:03:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 46% [15:03:33] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:48] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.015 second response time [15:04:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.26, 13.30, 11.71 [15:04:56] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:15] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:25] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.043 second response time [15:05:32] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.017 second response time [15:05:43] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.247 second response time [15:06:20] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:06:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 11.90, 11.41 [15:06:57] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.471 second response time [15:07:08] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:07:44] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.574 second response time [15:07:47] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8743 MB (22% inode=96%); [15:08:10] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.76, 13.13, 11.91 [15:09:22] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:21] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.032 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:10:25] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.015 second response time [15:11:30] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 47% [15:12:13] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.524 second response time [15:12:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.82, 10.20, 11.10 [15:13:07] PROBLEM - prometheus131 SSH on prometheus131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:34] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:13:37] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:01] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.009 second response time [15:14:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.275 second response time [15:14:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:38] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:41] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.65, 10.62, 11.07 [15:15:49] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.736 second response time [15:16:02] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:09] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:16:20] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [15:16:23] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw131 [15:16:39] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 40% [15:16:50] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [15:16:53] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:07] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8742 MB (22% inode=96%); [15:17:14] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:17:19] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:25] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [15:17:29] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.038 second response time [15:18:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 8.087 second response time [15:18:19] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.047 second response time [15:18:23] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.056 second response time [15:18:31] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:18:51] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.650 second response time [15:19:05] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:12] RECOVERY - prometheus131 SSH on prometheus131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:19:18] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.065 second response time [15:19:24] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:19:45] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.228 second response time [15:19:50] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:20:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [15:20:53] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.042 second response time [15:21:05] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:21:05] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [15:21:28] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.02, 0.00 [15:22:30] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:35] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:23:26] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:32] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 5.977 second response time [15:24:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:34] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.204 second response time [15:24:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.28, 0.10, 0.02 [15:25:13] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.019 second response time [15:25:38] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.779 second response time [15:26:33] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:26:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:26:59] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:27:08] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.023 second response time [15:27:31] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 25% [15:28:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:28:08] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [15:28:35] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:28:58] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.338 second response time [15:30:09] PROBLEM - wiki.beergeeks.co.il - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404435873031616 seconds [15:30:30] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.202 second response time [15:30:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 4.242 second response time [15:30:39] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [15:31:01] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4217 MB (10% inode=96%); [15:31:31] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.038 second response time [15:31:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:40] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:50] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:52] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 57% [15:33:31] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:34:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.04, 0.04, 0.00 [15:34:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:34:56] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:19] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.046 second response time [15:36:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.03, 0.01 [15:36:51] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 31% [15:36:52] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:37:32] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [15:37:34] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:38:31] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.041 second response time [15:39:13] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:30] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.247 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:39:32] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:35] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:41:28] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [15:41:29] PROBLEM - grc.repository.archiopedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - grc.repository.archiopedia.org All nameservers failed to answer the query. [15:41:31] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.073 second response time [15:41:38] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:00] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.017 second response time [15:42:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [15:43:42] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:44:42] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:42] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:55] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.114 second response time [15:46:08] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [15:46:13] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:46:52] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:47:07] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406909942626953 seconds [15:47:15] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.028 second response time [15:47:40] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.052 second response time [15:47:52] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:48:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:48:20] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:48:47] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:49:06] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4211 MB (10% inode=96%); [15:49:19] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.360 second response time [15:49:24] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:41] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:44] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:50] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.033 second response time [15:50:02] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.537 second response time [15:50:33] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [15:51:24] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.114 second response time [15:51:58] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.042 second response time [15:52:01] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [15:52:15] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:53:15] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:21] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.023 second response time [15:53:39] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:54:05] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.163 second response time [15:54:28] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0006897747517 secs [15:54:44] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 45% [15:54:48] PROBLEM - prometheus131 Current Load on prometheus131 is WARNING: WARNING - load average: 3.28, 3.70, 3.98 [15:54:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 10.66, 11.78 [15:55:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.294 second response time [15:55:50] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.014 second response time [15:57:42] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:58:07] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:12] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:56] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.040 second response time [15:59:03] RECOVERY - wiki.beergeeks.co.il - reverse DNS on sslhost is OK: SSL OK - wiki.beergeeks.co.il reverse DNS resolves to cp30.miraheze.org - CNAME OK [16:00:27] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:00:36] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:01:02] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:01:22] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:01:55] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:33] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.025 second response time [16:02:37] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 38% [16:02:39] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 38% [16:03:12] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 3.098 second response time [16:03:17] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.068 second response time [16:04:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.65, 10.85, 11.02 [16:06:03] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.043 second response time [16:06:03] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [16:06:21] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.192 second response time [16:06:45] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 10.51, 10.88 [16:07:28] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.583 second response time [16:08:53] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.02, 0.02 [16:08:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.22, 12.29, 11.48 [16:11:02] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: CRITICAL - load average: 4.69, 1.39, 0.49 [16:11:07] RECOVERY - grc.repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - grc.repository.archiopedia.org reverse DNS resolves to cp31.miraheze.org - CNAME OK [16:11:23] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:43] PROBLEM - prometheus131 PowerDNS Recursor on prometheus131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:12:06] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f6a5040fd768...782f089a7681 [16:12:19] [url] Comparing f6a5040fd768...782f089a7681 · miraheze/mw-config · GitHub | github.com [16:12:19] [02miraheze/mw-config] 07Universal-Omega 03782f089 - Add bastion to disallowed subdomains [16:12:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.58, 11.87, 11.55 [16:13:30] miraheze/mw-config - Universal-Omega the build passed. [16:13:44] RECOVERY - prometheus131 PowerDNS Recursor on prometheus131 is OK: DNS OK: 0.655 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:14:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/782f089a7681...a8d57b54b10d [16:14:02] [url] Comparing 782f089a7681...a8d57b54b10d · miraheze/mw-config · GitHub | github.com [16:14:03] [02miraheze/mw-config] 07Universal-Omega 03a8d57b5 - Remove invalid subdomain from disallow list [16:14:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:34] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39138 bytes in 0.034 second response time [16:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 11.69, 11.50 [16:14:58] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw131 mw142 [16:15:02] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:10] miraheze/mw-config - Universal-Omega the build passed. [16:15:25] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:36] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.078 second response time [16:15:39] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:15:53] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:01] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:28] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:48] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [16:16:56] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [16:17:19] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [16:17:40] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:52] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [16:18:32] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.403 second response time [16:18:59] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.269 second response time [16:20:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:20:12] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:36] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 1.561 second response time [16:20:39] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:20:44] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [16:20:53] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw132 [16:21:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:10] PROBLEM - cp30 Stunnel HTTP for test131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [16:21:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:21:36] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:39] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:57] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:15] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.654 second response time [16:22:30] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.975 second response time [16:22:46] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.047 second response time [16:22:49] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.914 second response time [16:22:53] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [16:23:35] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.038 second response time [16:24:09] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.048 second response time [16:24:21] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:24:34] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:25:07] RECOVERY - cp30 Stunnel HTTP for test131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.347 second response time [16:25:18] !log increase prometheus131 ram by 1g - https://phabricator.miraheze.org/T9599 [16:25:18] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 9.501 second response time [16:25:19] [url] ⚓ T9599 [Existing] Server Resource Request for prometheus | phabricator.miraheze.org [16:25:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:11] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.288 second response time [16:26:26] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 53% [16:26:30] !log [@test131] starting deploy of {'config': True} to all [16:26:31] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:26:33] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:33] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [16:27:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:39] PROBLEM - prometheus131 NTP time on prometheus131 is CRITICAL: connect to address 2a10:6740::6:402 port 5666: No route to hostconnect to host 2a10:6740::6:402 port 5666: No route to host [16:27:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:28:12] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:32] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.210 second response time [16:28:40] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 38% [16:28:40] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:28:43] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:55] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:33] RECOVERY - prometheus131 Current Load on prometheus131 is OK: OK - load average: 0.50, 0.21, 0.08 [16:29:41] RECOVERY - prometheus131 NTP time on prometheus131 is OK: NTP OK: Offset -0.001246213913 secs [16:30:04] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:06] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.377 second response time [16:30:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:41] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.336 second response time [16:30:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.350 second response time [16:30:54] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [16:31:04] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:31:24] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:31:48] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 2.314 second response time [16:32:22] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.044 second response time [16:32:26] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:45] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:01] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.092 second response time [16:33:14] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:31] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.051 second response time [16:33:36] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.071 second response time [16:34:33] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:36] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.058 second response time [16:34:41] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.018 second response time [16:34:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [16:36:20] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:32] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:36:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:36:59] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.055 second response time [16:37:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:38:04] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:12] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [16:38:12] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:38:46] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:23] !log [@mwtask141] starting deploy of {'config': True} to all [16:39:25] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw121.miraheze.org [16:39:44] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.571 second response time [16:40:08] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:14] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.068 second response time [16:40:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [16:40:48] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.015 second response time [16:41:01] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:21] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.271 second response time [16:41:53] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw131 mw132 mw142 [16:42:19] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [16:43:39] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:51] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [16:43:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:45:16] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.181 second response time [16:45:35] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:45:57] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.040 second response time [16:46:06] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.347 second response time [16:46:29] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406676530838013 seconds [16:46:56] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:47:01] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 8.163 second response time [16:47:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.036 second response time [16:47:33] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.551 second response time [16:47:38] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.429 second response time [16:48:47] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:02] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.05, 0.01 [16:49:13] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.083 second response time [16:49:52] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [16:50:32] [02mw-config] 07Universal-Omega synchronize pull request 03#4864: MirahezeFunctions: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/mw-config/pull/4864 [16:50:32] [url] Page not found · GitHub · GitHub | github.com [16:50:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2a70e23f94b4...b051b9cc6213 [16:50:34] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:34] [url] Comparing 2a70e23f94b4...b051b9cc6213 · miraheze/mw-config · GitHub | github.com [16:50:35] [02miraheze/mw-config] 07Universal-Omega 03b051b9c - Update MirahezeFunctions.php [16:50:43] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:51:36] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw121 mw122 [16:51:45] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [16:51:47] miraheze/mw-config - Universal-Omega the build passed. [16:52:21] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:32] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.787 second response time [16:53:09] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:54:36] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [16:54:52] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [16:55:25] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.033 second response time [16:55:52] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:33] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: connect to address localhost and port 8202: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:56:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: connect to address localhost and port 8104: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:56:50] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.329 second response time [16:56:50] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:57:07] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:08] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:57:13] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: connect to address localhost and port 8200: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:57:21] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: connect to address localhost and port 8107: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:57:33] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [16:57:41] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [16:57:42] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:57:57] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: connect to address localhost and port 8180: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:58:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:12] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:58:19] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:33] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:58:41] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:59:10] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [16:59:12] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:14] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.052 second response time [16:59:22] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:59:42] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:59:47] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [16:59:54] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3385 bytes in 0.025 second response time [16:59:56] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.717 second response time [16:59:59] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:00:07] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:00:30] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.030 second response time [17:00:59] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [17:01:10] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:01:29] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.067 second response time [17:01:34] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:02:12] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.301 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:02:20] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.070 second response time [17:03:12] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.050 second response time [17:03:28] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:34] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.016 second response time [17:03:47] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.235 second response time [17:03:52] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.056 second response time [17:04:42] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 45% [17:06:17] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:06:48] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:13] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [17:07:14] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.198 second response time [17:07:17] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.345 second response time [17:07:49] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:21] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [17:08:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:31] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [17:08:36] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.323 second response time [17:08:55] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.328 second response time [17:09:19] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.047 second response time [17:09:25] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.023 second response time [17:09:49] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.107 second response time [17:09:50] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.387 second response time [17:10:14] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [17:10:23] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 33% [17:10:25] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3397 bytes in 7.263 second response time [17:12:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:13:24] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:15:50] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.075 second response time [17:16:27] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:16:55] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:05] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:17:53] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:45] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 54% [17:18:59] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3396 bytes in 7.084 second response time [17:19:38] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:52] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.228 second response time [17:21:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:34] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.023 second response time [17:21:42] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:21:56] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.034 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:22:15] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:37] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:49] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:23:19] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3398 bytes in 7.222 second response time [17:23:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.704 second response time [17:23:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [17:23:56] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:19] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8719 MB (22% inode=96%); [17:24:47] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9146 MB (23% inode=96%); [17:25:08] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.749 second response time [17:25:14] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:55] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [17:26:02] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:02] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:26:36] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 33% [17:27:33] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.031 second response time [17:27:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:27:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:58] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [17:28:02] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [17:29:17] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3397 bytes in 3.068 second response time [17:29:34] PROBLEM - wiki.horizonsend.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.horizonsend.net' expires in 15 day(s) (Thu 25 Aug 2022 17:01:32 GMT +0000). [17:29:47] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:29:48] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:54] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:17] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:33] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:42] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 43% [17:31:44] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:31:50] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.554 second response time [17:31:55] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3397 bytes in 7.089 second response time [17:31:59] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:59] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:32:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5aef2b1d3164...fcbd18226d2b [17:32:05] [url] Comparing 5aef2b1d3164...fcbd18226d2b · miraheze/ssl · GitHub | github.com [17:32:06] [02miraheze/ssl] 07MirahezeSSLBot 03fcbd182 - Bot: Update SSL cert for wiki.horizonsend.net [17:32:15] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [17:32:31] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.321 second response time [17:32:34] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [17:33:23] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:40] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:26] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:34:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [17:34:27] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.059 second response time [17:34:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 42% [17:34:31] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:35:09] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 2.053 second response time [17:35:34] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.253 second response time [17:35:36] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8717 MB (22% inode=96%); [17:36:19] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:36:53] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.050 second response time [17:37:06] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:37:45] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [17:38:19] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:39:02] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:39:07] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.068 second response time [17:39:23] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:39:29] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:39:30] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 43% [17:39:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:24] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:28] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:57] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:40:59] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0004758238792 secs [17:41:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [17:41:57] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:41:58] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.035 second response time [17:42:21] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.085 second response time [17:42:25] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [17:42:31] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 7.284 second response time [17:42:49] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:42:56] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:43:06] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.107 second response time [17:44:55] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 22% [17:45:36] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:45:38] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 59% [17:46:01] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:47:33] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:37] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9141 MB (23% inode=96%); [17:47:53] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:57] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:58] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:14] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.035 second response time [17:48:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:35] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:48:59] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:59] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [17:49:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:49:49] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3396 bytes in 1.028 second response time [17:49:52] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.029 second response time [17:50:04] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.353 second response time [17:50:39] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 59% [17:50:58] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [17:51:16] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.03, 0.00 [17:51:23] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:58] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [17:52:11] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.043 second response time [17:52:45] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:52:46] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.051 second response time [17:52:59] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw121 mw122 mw131 mw141 [17:53:10] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:54:00] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.276 second response time [17:54:16] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:54] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [17:55:13] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [17:55:21] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 0.020 second response time [17:55:46] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.60, 9.84, 7.87 [17:56:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:14] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.985 second response time [17:56:14] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:17] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:56:20] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:40] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 1.988 second response time [17:56:41] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:20] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:40] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.56, 8.33, 7.56 [17:57:41] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:58:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3398 bytes in 3.053 second response time [17:58:23] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:27] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:58:27] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:58:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:44] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:58:46] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 34% [17:58:55] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [17:59:07] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 1.040 second response time [17:59:14] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:59:22] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.438 second response time [17:59:26] RECOVERY - wiki.horizonsend.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.horizonsend.net' will expire on Mon 07 Nov 2022 16:31:59 GMT +0000. [17:59:40] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.532 second response time [18:00:26] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.813 second response time [18:00:32] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9140 MB (23% inode=96%); [18:00:32] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.286 second response time [18:00:52] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [18:00:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [18:01:14] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.184 second response time [18:01:21] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 33% [18:02:32] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:03] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:03:54] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 45% [18:03:56] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw121 mw141 [18:04:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:04:48] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:53] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.041 second response time [18:05:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:07] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:13] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:15] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8710 MB (22% inode=96%); [18:05:54] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [18:06:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.082 second response time [18:06:07] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [18:06:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:06:45] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.931 second response time [18:06:52] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:04] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:07:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29521 bytes in 3.125 second response time [18:07:06] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 1.039 second response time [18:07:06] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.159 second response time [18:07:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 42% [18:07:25] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:28] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.067 second response time [18:07:31] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:38] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.15, 1.40, 0.88 [18:08:38] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [18:08:52] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 47% [18:09:06] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:23] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9137 MB (23% inode=96%); [18:09:32] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.020 second response time [18:09:38] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.83, 1.52, 0.99 [18:10:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [18:10:51] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:38] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.40, 1.46, 1.04 [18:12:11] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:12:59] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:13:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:14:45] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:15:07] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:21] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:15:22] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:15:34] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:34] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [18:16:08] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 40% [18:16:16] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:16:21] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 53% [18:17:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:17:24] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.643 second response time [18:17:26] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.759 second response time [18:17:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.280 second response time [18:17:32] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:34] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.034 second response time [18:17:55] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 5.059 second response time [18:18:02] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:14] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:18:16] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.070 second response time [18:18:23] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.044 second response time [18:19:05] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:42] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.423 second response time [18:19:43] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 1.041 second response time [18:19:55] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:02] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.084 second response time [18:20:37] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:21:18] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 30% [18:21:27] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:48] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:22:22] [02puppet] 07Universal-Omega opened pull request 03#2769: role::mediawiki: update default_value for gluster_volume - 13https://github.com/miraheze/puppet/pull/2769 [18:22:22] [url] Page not found · GitHub · GitHub | github.com [18:22:44] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [18:22:48] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:08] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.785 second response time [18:23:44] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:55] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [18:24:05] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:24:16] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.053 second response time [18:24:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:25:16] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.041 second response time [18:25:20] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 3.075 second response time [18:25:51] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [18:26:19] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.016 second response time [18:26:31] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.095 second response time [18:27:24] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [18:27:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 41% [18:27:30] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:27:47] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:28:09] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:11] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.039 second response time [18:29:40] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:54] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:29:58] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 37% [18:30:07] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:30:16] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.062 second response time [18:30:22] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:31:54] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [18:32:04] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.112 second response time [18:32:24] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005095303059 secs [18:32:25] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 7.099 second response time [18:32:25] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:30] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [18:32:39] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:47] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:51] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 57% [18:34:27] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.064 second response time [18:34:33] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 45% [18:34:35] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.079 second response time [18:34:42] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:50] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:35:05] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.036 second response time [18:35:14] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.036 second response time [18:35:48] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:01] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:52] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:03] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 35% [18:37:44] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:01] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 37% [18:38:09] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:38:15] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:08] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.034 second response time [18:40:09] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:16] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:19] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.017 second response time [18:41:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 3.058 second response time [18:42:06] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:42:13] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:42:29] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 50% [18:42:38] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [18:42:58] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:43:04] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:24] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.685 second response time [18:44:58] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.039 second response time [18:45:01] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.051 second response time [18:45:09] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.048 second response time [18:45:17] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406343221664429 seconds [18:46:00] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:46:03] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:07] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.048 second response time [18:46:20] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:28] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 36% [18:47:15] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:47:28] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 1.048 second response time [18:48:05] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [18:48:14] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:48:15] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.041 second response time [18:48:27] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:48:29] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [18:49:10] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:49:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:57] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.248 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [18:51:08] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9127 MB (23% inode=96%); [18:51:23] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.03, 0.07, 0.04 [18:51:26] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:51:51] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.294 second response time [18:52:13] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8699 MB (22% inode=96%); [18:52:15] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:38] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 34% [18:54:01] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:54:12] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.061 second response time [18:54:32] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:55:56] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:00] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [18:56:02] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.049 second response time [18:56:23] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:56:28] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:56:39] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:09] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.024 second response time [18:57:39] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [18:58:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:58:02] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 7.195 second response time [18:58:27] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw141 [18:58:28] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:28] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.488 second response time [18:58:34] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [18:58:46] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [18:59:05] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:59:07] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:35] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.337 second response time [18:59:47] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw122 mw131 mw141 mw142 [19:00:00] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:00:47] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.090 second response time [19:00:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.17, 7.52, 11.48 [19:01:01] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:01:05] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [19:01:07] PROBLEM - cp31 Stunnel HTTP for mon141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:01:31] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [19:01:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.23, 10.01, 11.95 [19:02:29] PROBLEM - cp31 Stunnel HTTP for reports121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:03:05] RECOVERY - cp31 Stunnel HTTP for mon141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 0.328 second response time [19:03:07] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:03:21] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:37] PROBLEM - cp31 Stunnel HTTP for mail121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:03:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [19:03:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.94, 11.74, 12.31 [19:03:59] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.082 second response time [19:04:23] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [19:04:30] RECOVERY - cp31 Stunnel HTTP for reports121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.491 second response time [19:04:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 1.84, 4.78, 9.46 [19:05:04] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 2.059 second response time [19:05:12] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:05:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:28] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:05:29] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:35] RECOVERY - cp31 Stunnel HTTP for mail121 on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.250 second response time [19:05:42] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:05:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.09, 9.91, 11.54 [19:06:28] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:06:32] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:32] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:52] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:59] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8694 MB (22% inode=96%); [19:07:35] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.302 second response time [19:07:36] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.073 second response time [19:07:42] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 22% [19:07:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:19] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw141 mw142 [19:08:28] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:08:38] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 7.284 second response time [19:08:48] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [19:09:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.848 second response time [19:09:23] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:23] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.107 second response time [19:09:33] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.423 second response time [19:09:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.12, 7.01, 10.07 [19:10:04] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:10:10] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [19:10:45] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [19:10:57] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:58] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:20] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.828 second response time [19:12:44] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:24] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:13:38] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 54% [19:14:12] The website is giving me a hard time again. [19:14:20] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [19:14:32] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:14:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [19:14:43] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.052 second response time [19:14:46] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 3.045 second response time [19:15:13] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:15:14] PROBLEM - cp30 Stunnel HTTP for puppet141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:15:20] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.012 second response time [19:15:40] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:15:59] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.017 second response time [19:15:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:27] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.517 second response time [19:16:38] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.031 second response time [19:16:58] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:17:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:08] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.343 second response time [19:17:14] RECOVERY - cp30 Stunnel HTTP for puppet141 on cp30 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.235 second response time [19:17:23] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:35] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.331 second response time [19:17:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:17:48] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:17:50] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:17:56] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:18:46] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [19:18:52] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.035 second response time [19:19:12] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.054 second response time [19:19:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:35] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.037 second response time [19:19:43] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:19:48] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.070 second response time [19:19:53] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:20:03] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:45] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:47] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.761 second response time [19:21:13] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 0.020 second response time [19:21:38] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:21:43] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:21:47] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw142 [19:21:56] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.165 second response time [19:21:58] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 37% [19:22:01] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.triplescripts.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:22:02] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:05] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.081 second response time [19:22:13] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:22:18] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:22:21] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:22:27] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.020 second response time [19:22:34] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:22:40] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.450 second response time [19:22:45] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [19:23:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.405 second response time [19:23:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [19:23:39] PROBLEM - cp31 Stunnel HTTP for test131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:23:43] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.325 second response time [19:23:48] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.052 second response time [19:23:59] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.566 second response time [19:24:10] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.020 second response time [19:24:24] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:26] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:24:26] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:24:47] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:25:35] RECOVERY - cp31 Stunnel HTTP for test131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.328 second response time [19:25:40] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [19:26:22] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:26:28] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.677 second response time [19:26:28] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.307 second response time [19:26:53] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:26:53] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 23% [19:26:57] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39178 bytes in 0.039 second response time [19:27:13] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:27:16] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:24] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.469 second response time [19:27:38] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 36% [19:28:57] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:29:15] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.151 second response time [19:29:21] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [19:29:36] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:39] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [19:29:40] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:45] PROBLEM - cp31 Stunnel HTTP for mon141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:29:52] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:29:57] PROBLEM - cp31 Stunnel HTTP for puppet141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:30:32] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:30:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 2.211 second response time [19:30:48] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.761 second response time [19:30:56] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.835 second response time [19:30:59] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.270 second response time [19:31:31] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [19:31:44] RECOVERY - cp31 Stunnel HTTP for mon141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.362 second response time [19:31:56] RECOVERY - cp31 Stunnel HTTP for puppet141 on cp31 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.240 second response time [19:31:58] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 51% [19:32:06] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:59] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:33:15] PROBLEM - cp30 Stunnel HTTP for test131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:34:03] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [19:34:13] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:34:30] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [19:34:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.303 second response time [19:34:56] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:34:56] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:35:01] PROBLEM - cp31 Stunnel HTTP for matomo131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:35:12] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:35:15] RECOVERY - cp30 Stunnel HTTP for test131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.326 second response time [19:35:17] PROBLEM - cp31 Stunnel HTTP for mail121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:35:19] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 7.164 second response time [19:35:19] PROBLEM - cp31 Stunnel HTTP for test131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:35:36] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.230 second response time [19:35:56] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:35:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 0.019 second response time [19:36:11] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.715 second response time [19:36:16] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.046 second response time [19:36:51] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.019 second response time [19:36:56] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.707 second response time [19:37:02] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 17% [19:37:13] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.343 second response time [19:37:15] RECOVERY - cp31 Stunnel HTTP for mail121 on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.255 second response time [19:37:15] RECOVERY - cp31 Stunnel HTTP for test131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.321 second response time [19:37:17] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.348 second response time [19:37:17] PROBLEM - db111 APT on db111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:37:23] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:31] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [19:37:47] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:37:58] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [19:39:01] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2289 MB (5% inode=96%); [19:39:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.62, 10.26, 6.95 [19:39:18] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (8 critical updates). [19:39:51] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:40:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.22, 10.94, 8.71 [19:41:17] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:41:28] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:12] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [19:42:25] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.46, 11.38, 9.17 [19:42:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.59, 9.82, 7.59 [19:43:20] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:23] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:24] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.085 second response time [19:43:27] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.016 second response time [19:43:58] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.247 second response time [19:44:31] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 7.255 second response time [19:44:52] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:03] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.033 second response time [19:45:19] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.040 second response time [19:46:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.92, 9.89, 9.06 [19:46:48] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:46:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.57, 10.60, 8.41 [19:47:45] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:09] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.037 second response time [19:49:41] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.132 second response time [19:50:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:50:02] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [19:50:06] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:50:10] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:50:29] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:51:27] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.052 second response time [19:51:48] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:52:05] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.174 second response time [19:52:16] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.025 second response time [19:52:25] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:52:26] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.279 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:53:42] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:54:00] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:54:56] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:00] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.053 second response time [19:56:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.51, 10.80, 9.78 [19:57:04] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.17, 9.95, 9.60 [19:59:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 3.064 second response time [19:59:30] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [19:59:32] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:00:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.99, 10.17, 9.75 [20:02:00] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.10, 0.10, 0.06 [20:02:25] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:03:06] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 2301 MB (5% inode=96%); [20:03:51] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:04:20] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:35] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [20:06:15] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 0.018 second response time [20:06:26] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.571 second response time [20:08:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.80, 13.83, 11.38 [20:09:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:10:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.83, 10.62, 9.88 [20:10:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 12.00, 10.99 [20:11:52] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 0.020 second response time [20:12:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.40, 11.52, 10.29 [20:13:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:14:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.57, 11.00, 10.24 [20:15:25] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:15:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.91, 11.56, 10.52 [20:16:00] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:15] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:00] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.triplescripts.org All nameservers failed to answer the query. [20:20:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.41, 12.54, 11.42 [20:21:29] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.044 second response time [20:22:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.34, 11.55, 11.18 [20:23:11] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.028 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [20:23:47] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.042 second response time [20:24:29] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 3.054 second response time [20:25:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.86, 11.37, 11.15 [20:26:00] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.36, 11.79, 11.25 [20:28:31] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:28:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.26, 11.62, 11.27 [20:29:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:30:03] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:31:42] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:31:54] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [20:32:20] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:32:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.83, 12.77, 11.73 [20:33:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.07, 11.70, 11.17 [20:34:09] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.045 second response time [20:34:16] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.053 second response time [20:35:13] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e35420974de3...c58c2306d426 [20:35:54] [url] Comparing e35420974de3...c58c2306d426 · miraheze/puppet · GitHub | github.com [20:35:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.12, 10.76, 10.90 [20:35:55] [02miraheze/puppet] 07paladox 03c58c230 - db: update dbcopy ssh key [20:37:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 3.040 second response time [20:38:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.61, 2.00 [20:38:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 11.15, 11.47 [20:39:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.57, 8.65, 10.02 [20:40:33] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:40:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.90, 12.67, 11.99 [20:41:31] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:31] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8681 MB (22% inode=96%); [20:42:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 10.28, 3.73, 2.62 [20:42:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 11.50, 11.65 [20:43:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.15, 3.90 [20:43:43] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 0.043 second response time [20:43:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.88, 10.68, 10.63 [20:44:07] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:45:41] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:46:03] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:20] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 9.44, 10.12 [20:48:13] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.045 second response time [20:49:15] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:49:56] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 7.133 second response time [20:49:58] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp31.miraheze.org - CNAME OK [20:50:34] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:51:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 2.59, 3.37 [20:52:33] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.039 second response time [20:56:17] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:58:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.42, 1.60, 1.97 [21:00:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:53] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.40, 11.05, 10.33 [21:02:29] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:02:55] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [21:02:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.01, 11.41, 11.23 [21:03:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f3d70a9b362c [21:03:26] [02miraheze/mw-config] 07paladox 03f3d70a9 - Don't create any new wikis on c4 (db141) temporarily [21:03:27] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:03:28] [url] Page not found · GitHub · GitHub | github.com [21:03:29] [02mw-config] 07paladox opened pull request 03#4865: Don't create any new wikis on c4 (db141) temporarily - 13https://github.com/miraheze/mw-config/pull/4865 [21:03:29] [url] Page not found · GitHub · GitHub | github.com [21:03:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a8d57b54b10d...2b09ff8f6bdc [21:03:33] [url] Comparing a8d57b54b10d...2b09ff8f6bdc · miraheze/mw-config · GitHub | github.com [21:03:34] [02miraheze/mw-config] 07paladox 032b09ff8 - Don't create any new wikis on c4 (db141) temporarily (#4865) [21:03:35] [02mw-config] 07paladox closed pull request 03#4865: Don't create any new wikis on c4 (db141) temporarily - 13https://github.com/miraheze/mw-config/pull/4865 [21:03:36] [url] Page not found · GitHub · GitHub | github.com [21:03:37] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:03:37] [url] Page not found · GitHub · GitHub | github.com [21:03:38] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [21:03:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 11.06, 10.40 [21:04:02] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:04:05] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:04:14] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [21:04:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:30] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 0.018 second response time [21:04:33] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.250 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [21:04:37] miraheze/mw-config - paladox the build passed. [21:04:46] miraheze/mw-config - paladox the build passed. [21:05:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.62, 10.40, 10.24 [21:06:07] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:06:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.34, 2.30, 2.09 [21:07:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:08:34] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.098 second response time [21:09:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.51, 3.15 [21:09:31] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:09:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [21:09:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.72, 11.14, 10.54 [21:10:44] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 2.90, 2.96 [21:11:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.31, 3.90 [21:11:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:55] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 11.37, 11.72 [21:13:10] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.023 second response time [21:13:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:13:45] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikipedia.yogfront.ooo All nameservers failed to answer the query. [21:14:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.91, 12.13, 11.93 [21:15:02] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 7.196 second response time [21:15:03] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.146 second response time [21:15:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.88, 5.43, 4.50 [21:17:06] !log increase db141 disk by 100g [21:17:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 3.80, 3.27 [21:17:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:19:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.37, 3.18 [21:19:39] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:20:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4858-paladox-patch-2 [+1/-0/±1] 13https://github.com/miraheze/mw-config/commit/59001d020ea8 [21:20:40] [02miraheze/mw-config] 07paladox 0359001d0 - Revert "Remove read only (#4858)" [21:20:41] [02mw-config] 07paladox created branch 03revert-4858-paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:20:41] [url] Page not found · GitHub · GitHub | github.com [21:20:43] [02mw-config] 07paladox opened pull request 03#4866: Revert "Remove read only" - 13https://github.com/miraheze/mw-config/pull/4866 [21:20:43] ... [21:20:51] [02mw-config] 07paladox synchronize pull request 03#4866: Revert "Remove read only" - 13https://github.com/miraheze/mw-config/pull/4866 [21:20:51] ... [21:20:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4858-paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/59001d020ea8...70b711a2daca [21:20:53] [url] Comparing 59001d020ea8...70b711a2daca · miraheze/mw-config · GitHub | github.com [21:20:54] [02miraheze/mw-config] 07paladox 0370b711a - Update databaseDb111Migration.txt [21:21:07] [02mw-config] 07paladox closed pull request 03#4866: Revert "Remove read only" - 13https://github.com/miraheze/mw-config/pull/4866 [21:21:08] [url] Page not found · GitHub · GitHub | github.com [21:21:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/mw-config/compare/2b09ff8f6bdc...86361289a521 [21:21:09] [url] Comparing 2b09ff8f6bdc...86361289a521 · miraheze/mw-config · GitHub | github.com [21:21:10] [02miraheze/mw-config] 07paladox 038636128 - Revert "Remove read only" (#4866) [21:21:12] [02miraheze/mw-config] 07paladox deleted branch 03revert-4858-paladox-patch-2 [21:21:13] [02mw-config] 07paladox deleted branch 03revert-4858-paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:21:13] [url] Page not found · GitHub · GitHub | github.com [21:21:16] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:21:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:33] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 17s [21:21:40] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 0.053 second response time [21:21:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:59] miraheze/mw-config - paladox the build passed. [21:22:19] miraheze/mw-config - paladox the build passed. [21:23:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.17, 4.74, 3.78 [21:26:49] !log [@test131] starting deploy of {'config': True} to all [21:26:50] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:27:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:46] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:29:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:31:53] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:34:18] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [21:38:28] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:08] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [21:42:42] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:43:22] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.041 second response time [21:43:27] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406368732452393 seconds [21:47:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 10.92, 11.95 [21:49:22] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/changeDBCluster.php --wiki=testwiki --file /home/salt-user/file1 --db-cluster=c5 (END - exit=0) [21:49:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:17] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:51:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.16, 12.20, 12.21 [21:52:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-4866-revert-4858-paladox-patch-2 [+0/-1/±1] 13https://github.com/miraheze/mw-config/commit/1af2c975ab60 [21:52:29] [02miraheze/mw-config] 07paladox 031af2c97 - Revert "Revert "Remove read only" (#4866)" [21:52:30] [02mw-config] 07paladox created branch 03revert-4866-revert-4858-paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:52:31] [url] Page not found · GitHub · GitHub | github.com [21:52:32] [02mw-config] 07paladox opened pull request 03#4867: Revert "Revert "Remove read only"" - 13https://github.com/miraheze/mw-config/pull/4867 [21:52:32] [url] Page not found · GitHub · GitHub | github.com [21:52:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/mw-config/compare/86361289a521...d25883645b88 [21:52:39] [url] Comparing 86361289a521...d25883645b88 · miraheze/mw-config · GitHub | github.com [21:52:40] [02miraheze/mw-config] 07paladox 03d258836 - Revert "Revert "Remove read only"" (#4867) [21:52:41] [02mw-config] 07paladox closed pull request 03#4867: Revert "Revert "Remove read only"" - 13https://github.com/miraheze/mw-config/pull/4867 [21:52:42] [url] Page not found · GitHub · GitHub | github.com [21:52:43] [02mw-config] 07paladox deleted branch 03revert-4866-revert-4858-paladox-patch-2 - 13https://github.com/miraheze/mw-config [21:52:43] ... [21:52:44] [02miraheze/mw-config] 07paladox deleted branch 03revert-4866-revert-4858-paladox-patch-2 [21:52:45] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:52:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:58] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [21:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:53:31] miraheze/mw-config - paladox the build passed. [21:53:41] miraheze/mw-config - paladox the build passed. [21:57:06] !log [@test131] starting deploy of {'config': True} to all [21:57:07] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:16] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [22:23:01] [02puppet] 07Universal-Omega opened pull request 03#2770: Cleanup hieradata - 13https://github.com/miraheze/puppet/pull/2770 [22:23:01] [url] Page not found · GitHub · GitHub | github.com [22:25:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.04, 9.97, 12.00 [22:41:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 10.58, 10.72 [22:42:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.96, 1.80, 1.99 [22:43:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 9.83, 10.40 [22:44:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.90, 2.00 [22:45:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.26, 9.12, 10.06 [22:47:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.09, 3.89 [22:48:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.31, 10.62, 11.99 [22:50:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.60, 11.54, 12.15 [22:51:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 10.98, 5.33, 4.48 [22:51:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.01, 11.31, 10.57 [22:53:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 10.74, 10.49 [22:57:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.30, 13.24, 11.46 [23:05:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 11.22, 11.50 [23:07:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.12, 11.81, 11.67 [23:09:55] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:33:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 9.72, 11.83 [23:38:50] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [23:39:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.37, 12.02, 11.94 [23:43:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 11.37, 11.79 [23:45:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.25, 3.96 [23:51:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.70, 3.90 [23:51:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.82, 10.98, 11.20 [23:53:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 11.11, 11.29 [23:55:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.73, 11.71, 11.49