[00:02:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.16, 3.92, 3.07 [00:06:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.53, 3.13 [00:07:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.81, 1.98 [00:08:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 4.11, 3.38 [00:12:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.86, 3.44 [00:13:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.69, 1.85, 1.92 [00:14:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.93, 3.49 [00:15:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 10.81, 11.75 [00:15:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.60, 1.82 [00:16:32] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.02, 1.69, 1.16 [00:17:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 1.85, 1.88 [00:18:32] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.00, 1.74, 1.24 [00:20:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.30, 3.63, 3.54 [00:22:32] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.38, 1.62, 1.31 [00:24:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.34, 3.15 [00:25:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.30, 1.79, 2.00 [00:26:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.84, 3.61 [00:27:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.30, 12.11, 11.75 [00:27:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 1.93, 2.03 [00:28:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.94, 3.44 [00:28:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.81, 3.64 [00:29:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.70, 1.89, 2.00 [00:31:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.62, 11.36, 11.57 [00:32:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.41, 3.34 [00:34:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.16, 3.26 [00:34:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 3.88, 3.71 [00:35:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.27, 1.69 [00:36:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.85, 3.71 [00:38:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.82, 3.52 [00:39:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.86, 2.07, 1.90 [00:39:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.82, 3.56 [00:40:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.47, 4.64, 4.03 [00:41:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.11, 13.28, 12.06 [00:41:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.50, 1.84, 1.83 [00:43:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.91, 3.63 [00:45:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.05, 10.92, 11.44 [00:45:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.96, 3.70 [00:48:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.84, 3.24, 3.67 [00:49:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.54, 12.47, 11.86 [00:49:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.59, 1.97, 1.87 [00:50:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 3.83, 3.82 [00:51:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.11, 1.70, 1.78 [00:52:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.83, 3.83 [00:53:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.79, 1.45, 1.68 [00:53:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.73, 3.68 [00:54:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.06, 3.90 [00:55:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.93, 3.78 [00:57:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.25, 2.38, 1.98 [00:57:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.82, 3.74 [00:58:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.73, 3.81 [00:59:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.58, 1.95, 1.87 [01:00:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.27, 4.76, 4.21 [01:00:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4864: MirahezeFunctions: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/mw-config/pull/4864 [01:00:17] [url] Page not found · GitHub · GitHub | github.com [01:00:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b051b9cc6213...5dec32a42139 [01:00:19] [url] Comparing b051b9cc6213...5dec32a42139 · miraheze/mw-config · GitHub | github.com [01:00:20] [02miraheze/mw-config] 07Universal-Omega 035dec32a - Update MirahezeFunctions.php [01:00:37] [02mw-config] 07Universal-Omega edited pull request 03#4864: MirahezeFunctions: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/mw-config/pull/4864 [01:00:38] [url] Page not found · GitHub · GitHub | github.com [01:01:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.74, 3.77 [01:01:35] miraheze/mw-config - Universal-Omega the build passed. [01:04:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.92, 4.00 [01:05:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.48, 1.58, 2.00 [01:06:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4827: add moderation email to managewiki settings - 13https://github.com/miraheze/mw-config/pull/4827 [01:06:38] [url] Page not found · GitHub · GitHub | github.com [01:06:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/92d55cc6124d...56f53a624eb9 [01:06:39] [url] Comparing 92d55cc6124d...56f53a624eb9 · miraheze/mw-config · GitHub | github.com [01:06:40] [02miraheze/mw-config] 07Universal-Omega 0356f53a6 - Fixes [01:07:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 2.93, 3.38 [01:07:35] [02mw-config] 07Universal-Omega reviewed pull request 03#4827 commit - 13https://github.com/miraheze/mw-config/pull/4827#discussion_r944988601 [01:07:35] [url] add moderation email to managewiki settings by Reception123 · Pull Request #4827 · miraheze/mw-config · GitHub | github.com [01:07:49] miraheze/mw-config - Universal-Omega the build passed. [01:07:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 1.88, 2.05 [01:08:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.97, 3.96 [01:09:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 10.79, 11.69 [01:09:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.79, 1.99 [01:10:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.51, 3.80 [01:11:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.88, 11.84, 11.92 [01:11:16] [02mw-config] 07Universal-Omega closed pull request 03#4788: Revert "Revert "setup entitySources for gratisdatawiki and co"" - 13https://github.com/miraheze/mw-config/pull/4788 [01:11:16] [url] Page not found · GitHub · GitHub | github.com [01:11:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4787-revert-4678-patch-73 [01:11:20] [02mw-config] 07Universal-Omega deleted branch 03revert-4787-revert-4678-patch-73 - 13https://github.com/miraheze/mw-config [01:11:20] [url] Page not found · GitHub · GitHub | github.com [01:11:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.56, 1.87, 1.79 [01:11:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.82, 3.80, 3.84 [01:13:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.34, 1.69, 1.74 [01:13:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.57, 2.00, 2.01 [01:15:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.80, 3.61 [01:15:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.92, 1.98 [01:17:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 11.67, 11.98 [01:17:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.35, 1.43, 1.62 [01:17:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.91, 3.09, 3.38 [01:17:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.33, 2.05, 2.02 [01:17:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.27, 3.04, 3.59 [01:19:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.60, 12.36, 12.20 [01:21:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.33, 1.71, 1.71 [01:21:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.83, 4.78, 3.97 [01:21:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.27, 4.94, 4.22 [01:22:53] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:25:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.80, 2.39, 1.99 [01:28:32] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.83, 1.57, 1.36 [01:29:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.53, 2.00, 1.93 [01:30:32] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.51, 1.52, 1.37 [01:31:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.54, 2.53, 2.13 [01:49:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.28, 3.90 [01:51:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.37, 4.17, 4.13 [01:52:18] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp31.miraheze.org - NS RECORDS OK [01:58:32] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.10, 1.70, 1.49 [02:02:32] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.80, 1.85, 1.61 [02:06:32] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.48, 1.67, 1.60 [02:07:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.75, 1.94 [02:07:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.44, 3.92 [02:09:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.46, 1.92, 1.97 [02:11:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.75, 3.93 [02:13:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.38, 3.77 [02:19:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [02:19:58] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:53] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [02:23:14] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 50% [02:23:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:23:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 2.90, 3.32 [02:27:13] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 24% [02:27:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 2.88, 3.82 [02:33:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.33, 1.67, 1.93 [02:35:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.11, 2.13, 2.06 [02:35:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.68, 2.41, 3.31 [02:41:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.98, 1.95, 1.99 [02:43:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 2.21, 2.08 [02:45:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.58, 1.94, 1.99 [02:47:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.98, 2.00 [02:53:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.61, 1.82, 1.94 [03:05:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.50, 1.28, 1.63 [03:20:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.16, 2.62, 2.04 [04:16:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e27832fd2b54...8e2d27d99b2d [04:16:05] [url] Comparing e27832fd2b54...8e2d27d99b2d · miraheze/mw-config · GitHub | github.com [04:16:06] [02miraheze/mw-config] 07paladox 038e2d27d - Depool cp20/21 from wgCdnServers [04:16:41] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [04:16:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:16:55] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw142.miraheze.org [04:16:59] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [04:17:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:10] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 10s [04:17:10] miraheze/mw-config - paladox the build passed. [04:17:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.62, 1.58, 1.93 [04:17:24] !log [paladox@mwtask141] starting deploy of {'config': True, 'force': True} to all [04:17:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:33] !log [paladox@mwtask141] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 9s [04:17:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:08] !log reboot mw142 - seems http went down (seeing prometheus-phpfpm-exporter[339]: 2022/08/13 04:18:07 http: Accept error: accept tcp [::]:9253: accept4: too many open files; retrying in 1s" and very little cpu usage. [04:19:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.79, 2.54, 2.22 [04:20:22] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.339 second response time [04:20:24] PROBLEM - mw142 NTP time on mw142 is CRITICAL: NTP CRITICAL: Offset -0.6649634838 secs [04:20:25] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.292 second response time [04:20:32] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.322 second response time [04:21:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [04:21:45] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [04:22:27] RECOVERY - mw142 NTP time on mw142 is OK: NTP OK: Offset 0.002280563116 secs [04:23:16] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [04:23:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 3.00, 2.39 [04:24:30] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [04:25:05] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [04:25:16] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [04:25:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.13, 2.51 [04:26:37] !log [@test131] starting deploy of {'config': True} to all [04:26:38] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:26:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:26:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:27:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.40, 2.67 [04:29:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.28, 2.72 [04:29:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.45, 2.84 [04:31:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.16, 4.92, 3.47 [04:33:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.84, 3.11 [04:35:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.64, 3.51, 3.27 [04:37:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.47, 2.79, 2.87 [04:37:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.32, 2.81, 3.04 [04:39:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 9.76, 11.75 [04:43:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.25, 1.73, 1.97 [04:44:32] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw131 mw142 [04:45:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.58, 1.93, 2.00 [04:47:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.71, 1.91 [04:49:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 1.96, 1.98 [04:51:04] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw132 [04:51:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.09, 7.79, 9.76 [04:51:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.91, 1.96 [04:52:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.29, 9.30, 11.56 [04:53:00] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [04:54:41] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [04:58:33] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw122 mw141 [04:59:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 1.89, 1.88 [04:59:34] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:00:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.37, 10.37, 10.04 [05:00:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 10.92, 11.25 [05:00:31] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:01:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.23, 1.63, 1.79 [05:01:33] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.066 second response time [05:01:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.44, 9.93, 9.91 [05:02:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.58, 2.25, 2.01 [05:03:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [05:04:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:05:15] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw121 mw131 mw141 [05:05:27] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw121 mw141 [05:05:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.42, 3.28 [05:05:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.79, 11.10, 10.39 [05:06:23] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.69, 3.95, 3.46 [05:08:49] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:57] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.68, 3.43 [05:09:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.16, 12.19, 10.89 [05:11:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.28, 3.31 [05:12:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:12:46] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:13:00] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:13:35] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:14:13] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.813 second response time [05:14:18] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [05:14:31] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 17% [05:15:12] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [05:17:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:17:42] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [05:17:42] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [05:18:26] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:18:52] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:20:05] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:20:29] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:33] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.643 second response time [05:22:04] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.058 second response time [05:22:14] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:22:34] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:41] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.285 second response time [05:23:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 11.15, 11.28 [05:23:55] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:23:57] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:24:16] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.064 second response time [05:24:36] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.086 second response time [05:25:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 4.26, 3.49 [05:25:50] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:25:52] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11555 MB (30% inode=96%); [05:26:45] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:27:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.78, 13.84, 12.30 [05:27:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.16, 5.19, 3.60 [05:28:50] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.098 second response time [05:29:00] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.101 second response time [05:30:14] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:17] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:30:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8e2d27d99b2d...e8096627c8ee [05:30:38] [url] Comparing 8e2d27d99b2d...e8096627c8ee · miraheze/mw-config · GitHub | github.com [05:30:39] [02miraheze/mw-config] 07Universal-Omega 03e809662 - non-profit -> not-for-profit [05:31:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 11.84, 11.90 [05:31:46] miraheze/mw-config - Universal-Omega the build passed. [05:31:56] [02puppet] 07Universal-Omega opened pull request 03#2780: README: non-profit -> not-for-profit - 13https://github.com/miraheze/puppet/pull/2780 [05:31:56] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:31:57] [url] Page not found · GitHub · GitHub | github.com [05:32:12] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [05:32:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [05:33:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.09, 12.75, 12.22 [05:33:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.96, 3.55 [05:34:12] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:34:12] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.058 second response time [05:34:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:35:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.80, 3.86 [05:35:34] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.377 second response time [05:36:11] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:36:14] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.043 second response time [05:36:51] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:37:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.97, 3.91 [05:37:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.77, 4.06, 3.66 [05:37:43] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:45] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:38:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [05:38:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:05] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.52, 3.75 [05:39:23] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [05:39:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.76, 3.61 [05:39:39] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:40:04] !log [@mwtask141] starting deploy of {'config': True} to all [05:40:13] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [05:40:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:17] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3402 bytes in 0.020 second response time [05:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:40:52] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:40:52] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:41:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.33, 2.85, 3.29 [05:41:52] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:42:25] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:42:25] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:42:39] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:42:41] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.051 second response time [05:43:28] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:43:59] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:44:25] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.020 second response time [05:45:02] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:45:08] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:45:25] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.061 second response time [05:46:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [05:46:44] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:46:49] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.047 second response time [05:47:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 2.66, 3.27 [05:47:08] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.083 second response time [05:47:10] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:47:13] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.039 second response time [05:47:35] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.273 second response time [05:47:54] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:48:24] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.056 second response time [05:49:43] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:56] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.132 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [05:50:21] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:51:12] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 41% [05:51:32] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.051 second response time [05:51:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 7.291 second response time [05:51:40] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.048 second response time [05:51:47] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.077 second response time [05:52:00] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:28] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.025 second response time [05:52:48] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.198 second response time [05:53:07] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:11] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 21% [05:54:02] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:54:37] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.432 second response time [05:55:36] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:55:57] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:56:01] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [05:56:17] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:56:44] !log [@test131] starting deploy of {'config': True} to all [05:56:45] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:56:46] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:56:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:57:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:29] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:58:31] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.049 second response time [05:58:39] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:44] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001508027315 secs [05:59:18] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:00:10] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:15] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:00:29] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.045 second response time [06:00:53] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:26] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:47] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.096 second response time [06:02:31] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:36] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 0.021 second response time [06:03:40] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:04:10] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:04:27] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [06:04:33] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:04:38] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:23] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:05:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:06:11] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.181 second response time [06:06:21] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:42] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:08:14] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:08:44] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:08:49] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [06:08:56] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.066 second response time [06:09:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.69, 10.44, 11.84 [06:09:23] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.237 second response time [06:10:09] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.040 second response time [06:10:21] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:27] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:10:32] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:11:16] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.037 second response time [06:11:42] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:11:55] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.028 second response time [06:12:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.18, 9.01, 11.73 [06:12:25] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [06:12:31] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:45] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.215 second response time [06:13:07] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:13:54] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:13:56] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.077 second response time [06:14:12] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:28] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:14:34] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:06] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.026 second response time [06:15:11] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.065 second response time [06:15:49] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:16:36] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:00] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:31] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:18:02] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [06:18:12] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.035 second response time [06:18:33] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:18:36] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.018 second response time [06:18:50] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:18:57] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.033 second response time [06:19:11] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:19:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 8.36, 10.19 [06:19:33] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.035 second response time [06:19:37] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11420 MB (29% inode=96%); [06:20:05] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.91, 7.73, 10.04 [06:21:08] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:21:09] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:25] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.281 second response time [06:21:39] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.060 second response time [06:22:01] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:06] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:22:11] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:22:19] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.098 second response time [06:22:29] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:22:56] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:22:57] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [06:23:14] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.055 second response time [06:23:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:23:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [06:23:51] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.142 second response time [06:24:05] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.022 second response time [06:24:10] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.068 second response time [06:24:19] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:24:28] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:40] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [06:24:52] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:53] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:24:55] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:00] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.041 second response time [06:25:01] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:25:12] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:25:15] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:25:27] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:26:24] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 1.017 second response time [06:26:26] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:26:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:49] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:26:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 53% [06:26:53] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.181 second response time [06:27:01] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:27:03] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.128 second response time [06:27:07] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:27:14] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.090 second response time [06:27:15] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.121 second response time [06:27:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [06:27:26] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.049 second response time [06:28:06] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:59] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.161 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [06:29:12] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001655519009 secs [06:29:27] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:46] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:29:50] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.069 second response time [06:30:16] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 25% [06:30:22] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:30:30] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.108 second response time [06:30:42] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [06:30:42] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:31:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:31:32] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:31:42] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.046 second response time [06:31:51] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.052 second response time [06:31:57] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 28% [06:31:59] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:13] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:18] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.050 second response time [06:32:20] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11556 MB (30% inode=96%); [06:32:24] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [06:32:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:46] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:32:52] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:33:06] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:33:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.68, 1.76, 2.00 [06:33:33] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [06:34:38] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.031 second response time [06:34:42] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:01] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:09] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [06:35:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [06:35:20] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:51] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:53] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:35:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:36:04] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:36:25] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [06:36:31] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.282 second response time [06:36:38] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.048 second response time [06:36:58] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.050 second response time [06:37:01] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.092 second response time [06:37:01] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:37:08] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:27] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.136 second response time [06:37:44] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:47] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.019 second response time [06:37:49] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.164 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [06:37:53] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:00] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [06:38:14] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:38:27] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:59] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:00] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:33] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.088 second response time [06:39:53] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.043 second response time [06:39:59] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:40:21] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:40:43] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.321 second response time [06:40:49] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.044 second response time [06:41:05] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:41:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [06:41:57] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.026 second response time [06:42:09] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [06:42:49] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.030 second response time [06:43:01] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:43:26] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:43:46] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:13] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.084 second response time [06:44:16] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:44:32] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.069 second response time [06:44:48] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:48] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.149 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [06:45:04] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:45:08] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:15] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [06:45:28] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:45:37] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:01] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:02] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:46:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.304 second response time [06:47:09] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.248 second response time [06:47:09] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:15] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 42% [06:47:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.59, 1.70, 1.79 [06:47:24] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11553 MB (30% inode=96%); [06:47:25] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:25] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:29] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 44% [06:47:53] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:47:58] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [06:48:00] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:48:02] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 7.227 second response time [06:48:02] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:48:15] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 7.172 second response time [06:48:52] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:10] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005169510841 secs [06:49:12] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:40] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.038 second response time [06:50:01] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [06:50:05] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.117 second response time [06:50:10] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:50:19] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:24] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:50:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:50:30] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:50:31] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.030 second response time [06:50:33] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [06:50:48] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:51:11] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.074 second response time [06:51:30] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.029 second response time [06:51:38] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.015 second response time [06:52:08] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 31% [06:52:13] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.289 second response time [06:52:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.08, 0.04, 0.01 [06:52:37] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.017 second response time [06:52:47] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:59] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [06:53:17] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:53:22] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [06:54:02] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:54:07] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:54:23] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 0.042 second response time [06:55:29] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:32] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:55:49] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:55:58] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.027 second response time [06:56:06] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:56:16] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:22] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.073 second response time [06:56:26] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:56:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 10.25, 9.55 [06:57:00] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [06:57:29] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:57:58] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:58:04] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 43% [06:58:17] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.136 second response time [06:58:38] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [06:58:44] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:58:57] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.046 second response time [06:59:07] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.367 second response time [06:59:21] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:59:29] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:59:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.85, 10.79, 9.55 [06:59:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:00:04] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:00:12] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 41% [07:00:13] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:00:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.07, 10.72, 9.90 [07:00:33] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [07:00:36] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:01:02] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:01:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.38, 1.90, 1.96 [07:01:25] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:01:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.17, 9.83, 9.35 [07:01:43] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [07:01:45] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.042 second response time [07:01:56] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.021 second response time [07:01:58] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:02] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:15] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 32% [07:02:29] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [07:02:46] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:03:12] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.52, 2.10, 2.02 [07:03:22] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [07:03:25] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.082 second response time [07:03:42] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.051 second response time [07:04:34] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:04:40] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:04:41] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:04:43] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [07:04:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [07:04:56] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:04] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.453 second response time [07:05:09] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001794040203 secs [07:05:11] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:12] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.34, 13.01, 10.65 [07:05:38] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.098 second response time [07:05:50] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:06:15] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:06:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.57, 11.42, 10.33 [07:06:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.01, 0.00 [07:06:46] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:07:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:35] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 39% [07:08:14] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [07:08:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:08:55] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:08:58] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:19] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 11.88, 10.80 [07:09:40] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:42] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:53] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:09:59] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:10:00] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:10:21] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:10:24] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.050 second response time [07:10:26] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 1.026 second response time [07:10:34] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.023 second response time [07:10:48] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.068 second response time [07:10:52] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.322 second response time [07:11:11] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:29] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:43] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:45] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:11:45] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11424 MB (29% inode=96%); [07:11:54] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:11:56] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11548 MB (30% inode=96%); [07:12:04] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.310 second response time [07:12:37] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.221 second response time [07:12:40] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.026 second response time [07:12:46] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [07:12:49] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [07:13:07] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.121 second response time [07:13:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.68, 11.37, 10.80 [07:13:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 10.71, 10.60 [07:14:09] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [07:14:10] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.036 second response time [07:14:12] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.219 second response time [07:14:50] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:12] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:17] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:15:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 11.25, 10.82 [07:15:34] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.081 second response time [07:15:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:16:23] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [07:16:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.017 second response time [07:17:02] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.199 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:17:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.76, 1.99 [07:17:23] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.02, 0.03, 0.00 [07:18:03] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:18:13] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [07:18:59] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:19:15] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:19:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.94, 2.02 [07:19:30] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.018 second response time [07:19:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.34, 11.32, 10.88 [07:20:01] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:20:09] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.098 second response time [07:20:16] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.778 second response time [07:20:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:20:57] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.039 second response time [07:21:11] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.67, 11.69, 10.93 [07:21:15] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:21:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.88, 1.99 [07:21:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 45% [07:21:25] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.048 second response time [07:21:37] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.043 second response time [07:21:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.46, 10.83, 10.74 [07:21:52] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:22:15] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:41] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:22:43] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:44] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.020 second response time [07:23:06] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:23:10] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 1.017 second response time [07:23:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 11.45, 10.95 [07:23:16] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:23:17] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:23:19] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.022 second response time [07:23:20] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:23:21] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.095 second response time [07:23:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.00, 2.14, 2.08 [07:23:49] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:23:57] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:24:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.07, 0.02, 0.00 [07:24:21] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.208 second response time [07:24:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:24:25] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.052 second response time [07:24:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:55] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.049 second response time [07:25:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.86, 11.53, 11.01 [07:25:12] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [07:25:15] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [07:25:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:25:41] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:25:56] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.017 second response time [07:26:00] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [07:26:06] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:26:20] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.317 second response time [07:26:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 43% [07:26:24] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:26:36] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:27:05] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.143 second response time [07:27:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.63, 11.10, 10.89 [07:27:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 9.08, 9.98 [07:27:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.030 second response time [07:28:11] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 7.222 second response time [07:28:26] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 41% [07:28:46] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:28:47] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.156 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:29:08] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.100 second response time [07:29:11] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.091 second response time [07:29:22] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:29:29] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:29:35] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:05] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:11] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:30:32] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:30:53] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [07:31:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:31:22] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 30% [07:31:34] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.017 second response time [07:31:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:02] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:32:09] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:32:13] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:32:28] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:32:40] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.11, 11.17, 10.82 [07:33:04] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:10] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.15, 10.01, 10.03 [07:33:20] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.042 second response time [07:33:29] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [07:33:31] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:35] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [07:33:56] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:56] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:34:01] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:34:12] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.004171520472 secs [07:34:13] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:36] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.049 second response time [07:35:03] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.056 second response time [07:35:15] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:36] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.040 second response time [07:35:50] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:35:54] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:36:49] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.221 second response time [07:37:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.62, 12.79, 11.04 [07:37:15] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.202 second response time [07:37:16] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:37:26] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.048 second response time [07:38:01] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:38:08] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [07:38:10] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:38:26] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:38:59] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [07:39:13] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:18] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:53] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:39:54] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.016 second response time [07:39:55] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.216 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:39:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.91, 3.00 [07:40:10] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.022 second response time [07:40:11] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:22] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.163 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:40:22] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0008306205273 secs [07:40:23] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:40:41] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:41:01] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:02] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:23] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.082 second response time [07:41:29] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [07:41:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.47, 2.94 [07:42:03] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 7.291 second response time [07:42:56] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [07:43:05] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 1.065 second response time [07:43:09] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.468 second response time [07:43:18] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.015 second response time [07:43:44] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [07:43:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.18, 2.89 [07:43:54] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:44:09] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:44:29] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:44:32] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 0.020 second response time [07:44:55] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:03] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:45:38] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [07:46:09] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.036 second response time [07:46:21] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:46:31] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11539 MB (29% inode=96%); [07:47:23] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [07:47:38] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:42] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:43] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:47:49] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 40% [07:47:56] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:48:37] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:44] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.126 second response time [07:49:11] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:28] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:44] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [07:49:47] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:52] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:55] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:59] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.01, 0.02, 0.00 [07:49:59] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001560509205 secs [07:50:04] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.048 second response time [07:50:11] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:35] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:50:49] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:06] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.028 second response time [07:51:12] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:13] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [07:51:45] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.103 second response time [07:51:46] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:02] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 7.169 second response time [07:52:07] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.044 second response time [07:52:20] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 46% [07:52:58] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.046 second response time [07:53:08] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.135 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [07:53:14] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:53:37] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [07:54:15] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:54:17] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:54:54] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:55:05] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.401 second response time [07:55:06] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:55:15] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:55:27] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [07:55:44] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [07:56:08] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [07:56:19] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:56:39] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:56:44] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:52] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:56:56] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [07:57:03] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.00194388628 secs [07:57:07] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.375 second response time [07:57:10] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.076 second response time [07:57:15] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:57:25] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:58:14] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:58:29] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:58:53] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 47% [07:59:54] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:00:02] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:00:12] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.127 second response time [08:00:21] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.140 second response time [08:00:48] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:01:10] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:13] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:47] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [08:02:22] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:02:30] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:02:39] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:02] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:32] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 36% [08:03:43] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:58] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [08:04:54] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:04:58] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [08:04:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:02] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [08:05:19] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.035 second response time [08:05:50] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.077 second response time [08:05:54] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11534 MB (29% inode=96%); [08:05:57] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.024 second response time [08:06:22] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:30] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:06:37] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [08:06:38] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.247 second response time [08:06:49] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:15] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:21] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 3.059 second response time [08:07:37] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:49] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:53] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:19] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:08:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 46% [08:09:02] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:45] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:50] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:09:51] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:09:56] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.262 second response time [08:10:01] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0004379451275 secs [08:10:07] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:19] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:10:21] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [08:11:10] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:11:22] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:11:39] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.133 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:11:41] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:48] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:12:24] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.057 second response time [08:12:33] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [08:12:39] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [08:12:56] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.199 second response time [08:13:21] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 37% [08:13:32] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:13:47] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [08:13:55] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:14:07] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:10] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.070 second response time [08:14:13] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.213 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:14:18] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:14:52] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:17] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [08:15:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:44] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35731 bytes in 1.061 second response time [08:15:52] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.102 second response time [08:16:02] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:13] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:16] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.052 second response time [08:16:24] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:30] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.213 second response time [08:16:51] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:53] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [08:17:13] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.182 second response time [08:17:29] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:17:30] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:17:55] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11414 MB (29% inode=96%); [08:18:06] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0006764531136 secs [08:18:30] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:18:31] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:18:42] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [08:18:44] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:53] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:19:13] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.029 second response time [08:19:23] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:19:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.014 second response time [08:20:22] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:34] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.080 second response time [08:20:37] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.160 second response time [08:20:45] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:21:10] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:21:26] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [08:21:33] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:21:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:00] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [08:22:44] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:45] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:52] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:53] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:23:14] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.015 second response time [08:23:24] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:23:40] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [08:23:41] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [08:23:50] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:24:12] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [08:24:44] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [08:24:56] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.030 second response time [08:25:47] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.106 second response time [08:25:50] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:04] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.116 second response time [08:26:05] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:06] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:26:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [08:26:40] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [08:26:51] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:26:54] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.03, 0.03, 0.01 [08:27:32] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:27:43] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11525 MB (29% inode=96%); [08:27:46] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:28:21] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:32] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.296 second response time [08:28:35] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.022 second response time [08:28:41] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [08:28:46] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:28:56] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [08:29:14] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:29:31] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:29:41] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.260 second response time [08:30:13] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.139 second response time [08:30:32] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:30:38] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 7.199 second response time [08:31:31] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [08:31:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 58% [08:31:46] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.321 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:31:49] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:09] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.015 second response time [08:32:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:25] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [08:32:42] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:47] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:32:56] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:11] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 37% [08:33:32] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [08:33:39] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:39] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:01] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:33] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:37] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:43] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:34:52] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:05] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:44] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.278 second response time [08:35:51] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:00] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:06] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:14] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:36:43] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:46] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:37:07] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.042 second response time [08:37:10] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:37:33] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:37:43] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.056 second response time [08:37:52] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:38:03] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [08:38:05] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [08:38:22] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:38:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:40] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:38:56] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.021 second response time [08:39:04] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 43% [08:39:06] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [08:39:17] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [08:39:19] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:39:34] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.356 second response time [08:39:35] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 5.074 second response time [08:39:39] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.024 second response time [08:39:39] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:39:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:18] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.318 second response time [08:40:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 11.01, 11.92 [08:40:51] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:40:54] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:40:54] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.044 second response time [08:41:15] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:22] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [08:41:40] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:41:59] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:42:38] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.131 second response time [08:42:50] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11410 MB (29% inode=96%); [08:43:44] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:53] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [08:43:54] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:02] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 54% [08:44:12] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.036 second response time [08:44:13] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.052 second response time [08:44:35] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.033 second response time [08:45:11] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:13] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.274 second response time [08:45:56] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.057 second response time [08:46:36] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:58] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:00] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.042 second response time [08:47:11] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:47:13] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:47:36] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:47:47] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:48:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.053 second response time [08:48:16] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.018 second response time [08:48:19] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.018 second response time [08:48:23] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.081 second response time [08:48:26] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.188 second response time [08:48:26] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:49:01] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:49:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 10.72, 11.96 [08:49:37] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.031 second response time [08:49:59] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [08:50:19] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:50:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.95, 11.16, 11.25 [08:50:32] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 3.065 second response time [08:50:47] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:51:07] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:51:10] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 57% [08:51:12] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 48% [08:51:13] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:28] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:42] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:51:52] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.035 second response time [08:51:59] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:04] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:22] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 10.89, 11.12 [08:52:31] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:53:27] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.358 second response time [08:53:29] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:53:57] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:07] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:09] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:54:13] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.037 second response time [08:54:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.54, 12.16, 11.59 [08:54:56] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [08:55:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.73, 12.26, 12.12 [08:55:15] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:55:59] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:15] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.034 second response time [08:56:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:56:28] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.039 second response time [08:56:50] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:54] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11519 MB (29% inode=96%); [08:56:56] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:57:09] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:57:22] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:57:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.415 second response time [08:57:31] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.069 second response time [08:58:11] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.142 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [08:58:49] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.056 second response time [08:58:58] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 50% [08:59:07] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.052 second response time [08:59:15] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [08:59:27] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:53] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:59:57] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:00:07] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:00:17] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:00:21] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.038 second response time [09:00:22] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:00] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [09:01:07] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:25] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.101 second response time [09:01:50] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [09:01:53] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:58] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.004911988974 secs [09:02:02] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:02:05] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.047 second response time [09:02:07] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11406 MB (29% inode=96%); [09:02:19] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.044 second response time [09:02:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.44, 2.93 [09:02:22] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.180 second response time [09:02:24] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.192 second response time [09:02:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:10] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:11] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.050 second response time [09:03:16] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [09:03:36] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:56] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [09:04:00] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.147 second response time [09:04:08] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 3.11, 2.88 [09:04:45] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:05] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:16] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:35] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.047 second response time [09:05:35] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.01, 0.02, 0.00 [09:05:43] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:04] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:25] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:35] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.209 second response time [09:06:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [09:06:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.21, 10.84, 11.76 [09:07:35] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:51] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:08:08] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.000387430191 secs [09:08:14] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.017 second response time [09:08:28] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:08:28] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:02] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.303 second response time [09:09:26] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:35] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:09:41] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:43] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:09:47] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 3.048 second response time [09:09:47] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [09:10:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:11:02] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.074 second response time [09:11:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.37, 11.57, 11.76 [09:12:03] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.041 second response time [09:12:13] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:12:39] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.084 second response time [09:12:46] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:12:49] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:07] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.07, 11.64, 11.82 [09:13:17] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:21] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:41] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:53] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:14:02] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:14:09] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:14:35] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:03] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.193 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:15:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.55, 13.62, 12.56 [09:15:44] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.046 second response time [09:15:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:07] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:17] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [09:16:34] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.038 second response time [09:16:58] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.046 second response time [09:17:04] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:08] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:38] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 7.161 second response time [09:17:38] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:51] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:18:11] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 1.064 second response time [09:18:24] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [09:18:38] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.376 second response time [09:19:06] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:19:32] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [09:20:38] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.069 second response time [09:20:43] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 42% [09:20:45] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [09:20:49] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [09:20:56] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:20:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:12] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.153 second response time [09:21:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 11.29, 11.89 [09:21:34] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.035 second response time [09:21:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:57] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.338 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:22:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.039 second response time [09:22:17] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:23:08] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:23:42] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:23:50] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:23:56] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [09:24:14] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:24:19] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:23] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:37] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.032 second response time [09:25:04] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.066 second response time [09:25:04] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.016 second response time [09:25:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.74, 1.52, 1.88 [09:25:39] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:26:08] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0002872347832 secs [09:26:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:17] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11508 MB (29% inode=96%); [09:26:18] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.152 second response time [09:26:23] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:48] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.053 second response time [09:27:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.26, 0.12, 0.04 [09:27:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.45, 12.75, 12.14 [09:27:20] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:39] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:27:43] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:27:50] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.229 second response time [09:28:01] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [09:28:57] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:29:27] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.011 second response time [09:29:34] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.032 second response time [09:30:09] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [09:30:14] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [09:30:16] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [09:30:30] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.206 second response time [09:30:48] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:30:49] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.234 second response time [09:31:01] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [09:31:02] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 1.043 second response time [09:31:18] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.194 second response time [09:31:30] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:31:32] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.039 second response time [09:31:55] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:32:11] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 57% [09:32:14] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:32:21] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:32:47] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:33:14] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [09:33:31] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.153 second response time [09:33:53] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:34:04] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:34:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:34:35] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [09:34:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:35:04] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.051 second response time [09:35:11] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:36:16] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.352 second response time [09:36:43] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:36:45] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:46] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:01] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:01] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:09] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.157 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:37:24] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 7.238 second response time [09:37:55] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:58] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:58] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:59] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:34] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:38:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:39:01] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.186 second response time [09:40:00] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:16] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 46% [09:40:21] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:25] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:46] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:07] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.39, 1.50, 1.68 [09:41:28] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:35] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:41:40] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.059 second response time [09:41:51] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:01] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.06, 0.05 [09:42:16] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:39] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:42:43] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:13] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:16] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:43:58] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:44:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.054 second response time [09:44:05] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:44:11] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.071 second response time [09:44:20] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.020 second response time [09:45:02] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.80, 10.43, 11.60 [09:45:27] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.049 second response time [09:45:51] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.046 second response time [09:45:53] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:45:55] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.056 second response time [09:46:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 10.00, 11.82 [09:47:03] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:16] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.036 second response time [09:47:37] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.033 second response time [09:47:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:54] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.033 second response time [09:47:59] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:01] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 7.220 second response time [09:48:02] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.133 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:48:31] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:45] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [09:48:58] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.095 second response time [09:48:58] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11396 MB (29% inode=96%); [09:49:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.30, 10.60, 11.30 [09:49:27] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.182 second response time [09:49:30] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:49:49] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:50:09] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.88, 2.33, 1.96 [09:50:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 10.78, 11.64 [09:50:29] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.167 second response time [09:50:30] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 2.029 second response time [09:51:16] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 49% [09:51:43] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:51:49] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.203 second response time [09:52:08] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [09:52:16] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 12.00, 11.88, 11.98 [09:52:56] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.211 second response time [09:53:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.31, 11.90, 11.74 [09:53:15] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:47] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:54:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.94, 1.89 [09:54:30] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:03] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0009417235851 secs [09:55:24] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:55:32] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:40] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:55:44] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:46] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.158 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [09:55:46] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.026 second response time [09:55:57] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:56:19] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:30] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:43] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.054 second response time [09:56:47] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.094 second response time [09:57:12] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:57:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.55, 12.31, 11.92 [09:57:40] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:57:48] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.170 second response time [09:57:49] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:57:54] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [09:57:57] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [09:57:58] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:06] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:07] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:17] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:18] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.66, 13.33, 12.37 [09:59:33] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [09:59:42] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.044 second response time [09:59:48] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:03] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [10:00:17] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.046 second response time [10:00:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 11.57, 11.84 [10:00:41] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11496 MB (29% inode=96%); [10:00:52] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:01:08] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:13] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [10:01:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.23, 10.69, 11.43 [10:01:59] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:52] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:57] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [10:03:20] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:03:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.02, 1.47, 1.69 [10:04:01] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:04:14] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:04:19] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:04:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 53% [10:04:49] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:05:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.057 second response time [10:05:24] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:05:54] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:05:56] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 5.098 second response time [10:06:02] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:13] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.017 second response time [10:06:16] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.04, 0.03 [10:06:25] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.217 second response time [10:06:49] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.059 second response time [10:06:56] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:56] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:07:21] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [10:07:37] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [10:07:39] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.032 second response time [10:07:52] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0001172721386 secs [10:08:00] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [10:08:20] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [10:08:43] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:56] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [10:09:07] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11392 MB (29% inode=96%); [10:09:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.96, 10.89, 11.03 [10:09:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:34] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 8.121 second response time [10:09:58] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 1.047 second response time [10:10:20] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.017 second response time [10:10:37] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:15] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:11:31] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.226 second response time [10:11:59] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:02] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.204 second response time [10:12:15] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:25] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.039 second response time [10:12:36] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:12:37] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:41] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:42] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:46] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.09, 11.15, 11.21 [10:13:30] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:13:44] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:13:55] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:14:19] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.078 second response time [10:14:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 50% [10:14:29] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.054 second response time [10:14:38] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.276 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:14:43] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.128 second response time [10:14:45] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [10:15:19] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:35] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:39] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:15:46] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.130 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:15:54] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:49] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.62, 2.02, 1.82 [10:17:34] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:48] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:17:50] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.06, 0.03, 0.00 [10:17:51] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.150 second response time [10:18:10] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [10:18:13] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.075 second response time [10:18:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.10, 8.05, 9.90 [10:18:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:18:45] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.025 second response time [10:18:53] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [10:19:34] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:20:22] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.091 second response time [10:21:00] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.067 second response time [10:21:13] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:21:32] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:21:32] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:39] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:21:47] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.110 second response time [10:21:57] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [10:22:05] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [10:22:05] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005102753639 secs [10:22:34] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:22:43] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.036 second response time [10:22:45] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.116 second response time [10:22:46] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.027 second response time [10:22:59] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.040 second response time [10:23:00] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:23:04] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:23:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.36, 9.16, 10.07 [10:23:28] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.025 second response time [10:23:31] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.143 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:23:34] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.067 second response time [10:23:47] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:24:47] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:24:56] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.086 second response time [10:25:04] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:25:23] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:24] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:34] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11487 MB (29% inode=96%); [10:25:44] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:25:48] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [10:25:57] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:18] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:38] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:45] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.393 second response time [10:26:53] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:11] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.08, 1.81, 1.93 [10:27:41] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.047 second response time [10:27:50] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.227 second response time [10:28:41] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:48] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:49] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:53] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:28:57] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:29:17] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:20] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:29:23] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.016 second response time [10:29:23] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.046 second response time [10:29:45] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.04, 0.08, 0.02 [10:29:46] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.274 second response time [10:30:02] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 58% [10:30:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:30:23] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.025 second response time [10:30:25] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:30:28] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:30:36] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [10:30:48] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.040 second response time [10:31:05] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:31:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.39, 9.50, 9.58 [10:31:10] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:31:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.11, 1.88, 1.93 [10:32:26] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:32:41] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:51] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:04] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [10:33:17] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:33:17] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:24] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 57% [10:33:25] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [10:33:34] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.281 second response time [10:33:44] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:53] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:34:01] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:01] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:09] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.267 second response time [10:35:16] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [10:35:21] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.287 second response time [10:35:30] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:33] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:46] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:20] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:36:40] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [10:36:52] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.09, 9.48, 9.68 [10:37:14] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:31] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:38:30] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 38% [10:38:44] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.297 second response time [10:39:04] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.033 second response time [10:39:21] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:22] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:39:33] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.036 second response time [10:39:55] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.328 second response time [10:39:58] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.027 second response time [10:41:17] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:41:30] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:41:45] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.026 second response time [10:41:48] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:04] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:07] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0004698336124 secs [10:42:19] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:42:22] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [10:42:43] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.223 second response time [10:42:50] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:43:42] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:43:49] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:43:51] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11381 MB (29% inode=96%); [10:44:09] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:44:10] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.047 second response time [10:44:22] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 48% [10:45:01] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:55] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:46:02] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:46:16] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.024 second response time [10:46:31] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.091 second response time [10:46:54] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:47:01] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [10:47:18] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:31] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001421302557 secs [10:48:05] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.637 second response time [10:48:06] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:12] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:48:20] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:48:41] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:49:44] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:18] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11481 MB (29% inode=96%); [10:50:19] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:40] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [10:50:40] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [10:50:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.62, 12.06, 10.40 [10:50:51] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.029 second response time [10:51:04] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:51:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:51:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.02, 2.34 [10:52:14] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 54% [10:52:25] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.22, 11.37, 10.33 [10:53:11] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:53:29] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:30] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.169 second response time [10:53:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.75, 2.33 [10:53:55] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:54:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.64, 9.78, 9.89 [10:54:53] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.021 second response time [10:55:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:55:37] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:56:11] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:56:44] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:18] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.016 second response time [10:57:19] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:28] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.148 second response time [10:57:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:48] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:50] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:05] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:58:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 10.88, 10.02 [10:58:12] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [10:58:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.47, 4.34, 3.36 [10:58:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:58:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.24, 11.16, 10.58 [10:58:55] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.022 second response time [10:59:05] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.913 second response time [10:59:19] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [10:59:35] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:08] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [11:00:57] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:59] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:59] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 7.169 second response time [11:01:22] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.034 second response time [11:01:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 9.38, 9.62 [11:02:04] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.153 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:02:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:19] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:02:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.09, 9.57, 10.08 [11:02:30] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.032 second response time [11:02:57] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001783668995 secs [11:03:26] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:34] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.043 second response time [11:03:38] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [11:03:38] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.084 second response time [11:03:38] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:03:49] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.031 second response time [11:03:50] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:11] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:04:18] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.066 second response time [11:04:42] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.023 second response time [11:04:52] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 53% [11:04:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:05:10] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:29] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:34] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:05:36] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11376 MB (29% inode=96%); [11:05:42] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.042 second response time [11:05:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.034 second response time [11:05:50] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:06:00] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.042 second response time [11:06:10] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:06:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.44, 3.51 [11:06:48] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.257 second response time [11:07:03] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:07:18] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.078 second response time [11:07:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [11:07:35] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:07:45] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.053 second response time [11:08:00] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:08:09] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [11:08:11] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.110 second response time [11:08:46] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:09:06] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.042 second response time [11:10:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 2.90, 3.28 [11:10:20] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:25] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.019 second response time [11:10:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [11:10:47] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:11] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.026 second response time [11:11:12] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:12:15] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:12:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:12:31] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.20, 11.79, 10.55 [11:12:39] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:12:42] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [11:12:54] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [11:12:55] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.042 second response time [11:13:06] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11473 MB (29% inode=96%); [11:13:07] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:17] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.009 second response time [11:13:41] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:13:49] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0003815591335 secs [11:14:07] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.201 second response time [11:14:20] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.015 second response time [11:14:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.53, 10.19, 10.14 [11:14:40] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.356 second response time [11:14:45] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:14:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:53] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [11:14:57] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:15:07] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.187 second response time [11:15:37] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [11:15:38] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.137 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:15:43] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:15:46] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:16:07] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:17:04] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:17:18] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:17:28] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:17:29] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:34] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:17:41] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:17:49] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:55] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:18:50] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:09] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:26] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [11:19:27] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.073 second response time [11:19:29] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [11:19:44] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.208 second response time [11:20:11] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:20:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:45] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [11:21:16] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:21:32] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:21:49] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:00] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [11:22:01] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:13] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.019 second response time [11:22:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.035 second response time [11:22:32] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:23:43] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:23:44] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:23:49] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [11:23:59] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.039 second response time [11:24:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.21, 10.06, 9.94 [11:24:22] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.186 second response time [11:24:28] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [11:24:30] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.069 second response time [11:24:34] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 5.204 second response time [11:25:00] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:25:08] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.140 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:25:33] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.078 second response time [11:25:55] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:26:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.81, 9.23, 9.68 [11:26:38] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:26:41] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:26:41] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:26:57] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.034 second response time [11:26:58] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:27:05] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.035 second response time [11:27:43] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.050 second response time [11:27:56] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.16, 10.39, 9.65 [11:28:20] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:24] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.031 second response time [11:28:37] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0007990896702 secs [11:28:49] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:49] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:28:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:12] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:32] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [11:29:35] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.042 second response time [11:29:44] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:29:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.42, 9.78, 9.53 [11:29:59] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.271 second response time [11:30:03] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:06] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.037 second response time [11:30:54] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:31:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.98, 1.30, 1.84 [11:31:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.05, 0.05 [11:32:01] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:32:12] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [11:32:30] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:32:33] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.217 second response time [11:32:56] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.113 second response time [11:33:53] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [11:34:33] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:34:49] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:35:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.83, 1.12, 1.66 [11:35:22] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.034 second response time [11:35:24] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:36:04] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:05] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:36:36] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11463 MB (29% inode=96%); [11:36:40] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 49% [11:36:50] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:37:01] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:37:09] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.145 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:37:25] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:37:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 57% [11:38:01] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.074 second response time [11:38:15] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:41] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:08] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:28] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:35] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:47] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:39:48] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:26] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:40:41] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:41:23] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:41:31] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:42:09] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:42:21] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:42:55] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.411 second response time [11:43:13] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [11:43:14] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.021 second response time [11:43:21] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001289337873 secs [11:43:22] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:43:30] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [11:43:35] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:43:38] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.324 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:44:08] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.347 second response time [11:45:02] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:45:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 55% [11:46:37] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.016 second response time [11:46:59] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.048 second response time [11:47:07] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.075 second response time [11:47:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.02, 0.00 [11:47:32] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:47:50] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.011 second response time [11:47:55] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:48:03] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:48:19] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [11:48:22] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:48:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:48:31] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [11:49:07] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:49:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.17, 1.92, 1.73 [11:49:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.02, 0.00 [11:49:59] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [11:50:03] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:07] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.028 second response time [11:50:10] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [11:50:15] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:19] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.065 second response time [11:50:26] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [11:50:35] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:59] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:51:11] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0005829632282 secs [11:51:17] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.046 second response time [11:51:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.97, 1.79 [11:51:40] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 7.193 second response time [11:51:40] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.236 second response time [11:51:49] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:03] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.065 second response time [11:52:05] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.067 second response time [11:52:13] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [11:52:23] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:43] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:44] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.00112298131 secs [11:52:54] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [11:53:04] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:15] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11456 MB (29% inode=96%); [11:53:20] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 55% [11:53:27] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.139 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:53:29] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.338 second response time [11:53:30] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:29] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:43] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [11:55:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.133 second response time [11:55:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.92, 9.72, 8.83 [11:55:28] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11365 MB (29% inode=96%); [11:55:32] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 7.167 second response time [11:55:41] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.019 second response time [11:55:46] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:56:09] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:34] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:39] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:53] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:57:01] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:05] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:57:14] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:57:21] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:57:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.41, 1.60, 1.67 [11:57:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.98, 9.99, 9.05 [11:57:29] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.512 second response time [11:57:30] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:58:12] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:16] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:58:25] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 55% [11:58:28] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.146 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:58:45] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:59:10] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:59:20] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.186 second response time [11:59:36] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [11:59:46] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:09] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:00:35] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 7.218 second response time [12:00:55] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.043 second response time [12:01:07] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [12:01:14] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 48% [12:01:15] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.02, 10.31, 9.40 [12:01:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:34] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.099 second response time [12:01:50] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:01:59] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:02:13] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [12:02:24] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.043 second response time [12:03:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.38, 2.22, 1.89 [12:03:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 47% [12:03:43] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:58] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:04:09] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:04:14] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:04:14] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [12:04:22] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:23] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:04:23] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.162 second response time [12:04:39] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [12:05:03] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.054 second response time [12:05:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.90, 10.20, 9.59 [12:05:14] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:05:24] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 1.027 second response time [12:05:24] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.208 second response time [12:05:42] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:06:02] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.058 second response time [12:06:08] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:06:15] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [12:06:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:06:26] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.345 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:06:29] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.223 second response time [12:06:58] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [12:08:06] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [12:08:11] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:08:30] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:09:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.54, 11.17, 10.09 [12:09:10] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:10] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [12:09:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.89, 11.60, 9.88 [12:09:24] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:36] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.036 second response time [12:09:49] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:09:53] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 7.167 second response time [12:10:06] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:17] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.245 second response time [12:10:27] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:55] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [12:11:03] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:11:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.38, 10.95, 10.14 [12:11:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:08] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:11:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:11:42] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:47] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 4.869699478e-05 secs [12:12:06] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:12:16] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:12:24] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.034 second response time [12:12:38] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 11.66, 10.50 [12:13:08] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:13:18] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.637 second response time [12:13:20] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:13:45] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.228 second response time [12:14:21] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.058 second response time [12:14:29] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [12:14:40] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11450 MB (29% inode=96%); [12:14:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 10.92, 10.39 [12:15:16] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:15:32] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:15:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.60, 3.00 [12:16:00] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.060 second response time [12:16:03] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:37] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [12:16:44] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [12:16:49] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.39, 9.92, 10.10 [12:16:57] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:17:07] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.303 second response time [12:17:21] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 47% [12:17:31] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:17:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 3.00, 2.85 [12:17:58] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [12:18:29] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:50] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 8.000 second response time [12:19:11] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:19:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.11, 10.68, 10.95 [12:19:28] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:19:40] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:43] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.206 second response time [12:20:29] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:20:35] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [12:20:58] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.057 second response time [12:21:03] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:21] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.084 second response time [12:21:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.019 second response time [12:21:34] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:21:43] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.035 second response time [12:22:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:29] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:39] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:40] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:41] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:53] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:23:06] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.95, 8.52, 9.98 [12:23:20] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.050 second response time [12:23:24] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:23:33] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [12:23:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:24:08] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.068 second response time [12:24:34] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:38] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.040 second response time [12:24:50] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:51] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [12:25:17] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:25:21] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001897603273 secs [12:25:26] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:25:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:19] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:26:38] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 49% [12:27:08] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.135 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:27:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.69, 2.00 [12:27:28] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11446 MB (29% inode=96%); [12:27:34] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:51] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:53] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:27:53] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:27:54] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:57] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:27:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.048 second response time [12:28:18] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [12:28:40] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:42] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [12:28:42] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.094 second response time [12:29:12] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.75, 1.98 [12:29:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:29:29] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [12:29:47] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:03] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:06] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.03, 0.00 [12:30:29] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:42] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:56] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 1.031 second response time [12:30:59] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:10] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11358 MB (29% inode=96%); [12:31:14] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:31:18] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.013 second response time [12:31:52] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [12:32:01] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:21] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:38] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:58] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:33:13] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.023 second response time [12:33:31] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:34:34] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:34:44] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:34:45] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:01] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [12:35:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.73, 1.90 [12:35:45] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 3.068 second response time [12:36:28] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 1.032 second response time [12:37:06] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [12:37:26] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:37:29] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 44% [12:37:33] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.057 second response time [12:37:38] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:51] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:05] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:20] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:38:21] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:38:49] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:54] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 57% [12:39:21] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.11, 2.07, 2.01 [12:39:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 10.76, 9.91 [12:39:57] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 1.038 second response time [12:40:05] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:40:11] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.187 second response time [12:40:14] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:40:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.13, 11.30, 10.35 [12:40:51] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11442 MB (29% inode=96%); [12:40:55] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:41:00] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [12:41:02] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:41:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.32, 1.84, 1.93 [12:41:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.47, 9.50, 9.55 [12:41:33] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:41:50] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:01] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 7.265 second response time [12:42:12] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:42:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:42:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 10.52, 10.18 [12:43:21] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 2.07, 2.01 [12:43:35] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.195 second response time [12:43:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:10] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0009655952454 secs [12:44:14] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:44:18] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.033 second response time [12:44:19] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.032 second response time [12:44:25] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:44:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 54% [12:44:29] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0001108646393 secs [12:44:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.86, 9.58, 9.87 [12:44:53] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.053 second response time [12:45:03] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:45:07] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [12:45:38] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.270 second response time [12:45:48] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.028 second response time [12:45:52] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [12:45:57] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:51] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:22] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:33] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:40] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.074 second response time [12:47:41] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:48:10] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.262 second response time [12:48:16] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.022 second response time [12:48:17] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:48:28] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.223 second response time [12:48:42] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:49:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [12:49:47] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.015 second response time [12:50:00] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:50:11] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:22] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:23] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.193 second response time [12:50:44] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.023 second response time [12:51:01] PROBLEM - wiki.helsinginpoytaroolipelaajat.fi - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.helsinginpoytaroolipelaajat.fi' expires in 15 day(s) (Mon 29 Aug 2022 12:44:24 GMT +0000). [12:51:10] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:51:49] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:51:51] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [12:52:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:11] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.030 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [12:52:14] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:52:15] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:34] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:35] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:52:38] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.076 second response time [12:52:47] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [12:52:47] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:53:07] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [12:53:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 10.45, 9.99 [12:53:16] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [12:53:49] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:54:45] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [12:54:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fbc02524c2f...902a26ca9e92 [12:54:46] [url] Comparing 9fbc02524c2f...902a26ca9e92 · miraheze/ssl · GitHub | github.com [12:54:47] [02miraheze/ssl] 07MirahezeSSLBot 03902a26c - Bot: Update SSL cert for wiki.helsinginpoytaroolipelaajat.fi [12:54:47] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:55:02] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [12:55:07] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11354 MB (29% inode=96%); [12:55:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.82, 11.64, 10.47 [12:55:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [12:55:27] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:55:40] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.213 second response time [12:55:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [12:55:49] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:55:51] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.207 second response time [12:55:55] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:55:57] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:55:59] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:20] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:40] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.016 second response time [12:56:52] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.07, 10.91, 10.35 [12:57:14] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:59] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:00] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:24] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:58:51] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [12:58:52] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.074 second response time [12:59:47] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:10] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:19] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:29] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:00:32] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:00:49] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:00] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.048 second response time [13:01:18] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.249 second response time [13:01:20] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 1.066 second response time [13:01:40] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:46] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:01:57] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:05] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.184 second response time [13:02:41] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.201 second response time [13:02:43] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:58] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:02:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.11, 10.97, 10.33 [13:03:01] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.178 second response time [13:03:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 11.14, 10.61 [13:03:17] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.275 second response time [13:03:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 55% [13:03:49] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:04:01] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [13:04:18] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.001059651375 secs [13:04:32] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.211 second response time [13:04:36] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.134 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:04:47] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:55] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:49] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.033 second response time [13:06:05] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.027 second response time [13:06:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:32] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.017 second response time [13:06:36] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:38] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:40] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [13:06:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.25, 9.99, 10.11 [13:06:53] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.17, 11.86, 11.18 [13:07:15] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.326 second response time [13:07:39] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.043 second response time [13:07:44] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 7.157 second response time [13:07:55] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:07:56] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:08:01] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.147 second response time [13:08:16] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.205 second response time [13:08:28] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:40] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.040 second response time [13:08:43] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 45% [13:08:48] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:53] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.031 second response time [13:09:04] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.087 second response time [13:09:05] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:29] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:10:05] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11434 MB (29% inode=96%); [13:10:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.779 second response time [13:10:38] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.125 second response time [13:10:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 20.70, 13.30, 11.29 [13:10:42] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:49] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 39% [13:10:55] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:10:56] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:11:21] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.036 second response time [13:11:23] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:10] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:26] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.083 second response time [13:12:40] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.061 second response time [13:12:44] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:12:53] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:13:29] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:13:33] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [13:13:36] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.052 second response time [13:14:04] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:14:20] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:25] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:14:35] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:14:53] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:15:07] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:15:20] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.94, 1.34, 1.93 [13:15:42] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:57] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 7.166 second response time [13:16:17] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.022 second response time [13:16:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 11.30, 11.16 [13:16:48] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.181 second response time [13:16:58] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:17:34] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:17:35] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.049 second response time [13:17:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.06, 0.06, 0.02 [13:17:49] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.039 second response time [13:18:03] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:18:32] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:18:39] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.204 second response time [13:18:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.41, 8.99, 9.91 [13:18:54] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11433 MB (29% inode=96%); [13:19:09] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:19:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 40% [13:19:24] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:19:26] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:19:31] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.032 second response time [13:19:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 0.017 second response time [13:19:49] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [13:19:55] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.249 second response time [13:20:03] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [13:20:07] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:10] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:57] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:04] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 43% [13:21:16] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:21:22] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.45, 2.68, 2.23 [13:21:32] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:21:41] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:22:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.78, 12.04, 11.40 [13:22:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.38, 2.72 [13:22:23] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.028 second response time [13:22:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:23:09] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:23:13] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:15] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.00, 0.00, 0.00 [13:23:18] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 35% [13:23:39] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -5.516409874e-05 secs [13:23:42] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:11] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:24:12] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.40, 2.83 [13:24:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.59, 3.23 [13:24:26] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:27] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.49, 12.22, 11.03 [13:24:35] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:43] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:25:00] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [13:25:24] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.061 second response time [13:25:26] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.079 second response time [13:25:27] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:25:27] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:03] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:11] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:26] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:43] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:44] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:46] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [13:26:57] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0007009804249 secs [13:27:06] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.37, 0.15, 0.05 [13:27:17] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:28] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.029 second response time [13:27:29] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 5.053 second response time [13:27:43] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.161 second response time [13:27:56] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.083 second response time [13:28:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.74, 5.03, 3.59 [13:28:22] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.158 second response time [13:28:33] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:37] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:00] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [13:29:21] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [13:29:23] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [13:29:41] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:54] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [13:30:17] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:19] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:30] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [13:30:35] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.025 second response time [13:30:38] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [13:30:45] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 7.284 second response time [13:31:29] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:35] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:48] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.186 second response time [13:31:50] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 0.040 second response time [13:32:17] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 49% [13:32:35] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:55] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.155 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:33:25] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.315 second response time [13:33:35] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:35] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:33:47] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:33:48] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.054 second response time [13:33:59] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [13:34:03] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:34:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.55, 3.62, 3.47 [13:34:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.90, 3.57 [13:34:28] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:34:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:32] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 51% [13:35:10] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.032 second response time [13:35:12] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.033 second response time [13:35:33] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.020 second response time [13:35:39] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.021 second response time [13:35:46] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11427 MB (29% inode=96%); [13:35:59] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.01, 3.14, 3.31 [13:36:27] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0007558166981 secs [13:36:29] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [13:36:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.07, 0.04 [13:36:44] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.045 second response time [13:37:25] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.243 second response time [13:37:27] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:37:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:37:33] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:37:37] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:42] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:38:45] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:22] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 7.217 second response time [13:39:35] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:43] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:44] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [13:40:03] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11345 MB (29% inode=96%); [13:40:52] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:07] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.354 second response time [13:41:08] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.664 second response time [13:41:17] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.024 second response time [13:41:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56% [13:41:38] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:43] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.263 second response time [13:41:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:50] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [13:41:51] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.245 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:42:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 4.46, 3.83 [13:43:20] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:43:30] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.075 second response time [13:43:45] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.165 second response time [13:43:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.51, 3.37 [13:43:52] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.058 second response time [13:44:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.82, 3.66 [13:44:08] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.043 second response time [13:44:27] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:44:33] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:36] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:45:06] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.82, 2.85, 3.14 [13:46:16] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:46:22] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [13:46:55] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:58] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:07] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 49% [13:47:14] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:24] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:44] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:46] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.153 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:47:58] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:19] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 5.077 second response time [13:48:40] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 24% [13:48:44] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:46] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [13:48:47] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:52] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:54] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:00] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:41] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [13:50:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.38, 3.12, 3.38 [13:50:02] RECOVERY - wiki.helsinginpoytaroolipelaajat.fi - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helsinginpoytaroolipelaajat.fi' will expire on Fri 11 Nov 2022 11:54:41 GMT +0000. [13:50:06] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:15] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:21] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:35] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:50:48] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.111 second response time [13:52:02] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11343 MB (29% inode=96%); [13:52:06] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 42% [13:52:07] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:12] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:52:17] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.02, 0.00 [13:52:21] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.072 second response time [13:52:23] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [13:52:25] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001723587513 secs [13:52:38] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:52:41] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:52:49] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:52:52] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:53:01] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:53:02] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.402 second response time [13:53:57] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.071 second response time [13:54:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.92, 9.70, 11.68 [13:54:39] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.020 second response time [13:54:39] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:54:50] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.035 second response time [13:55:02] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:05] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 10.53, 11.85 [13:55:17] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:25] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.045 second response time [13:55:40] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.180 second response time [13:55:59] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [13:56:02] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [13:56:13] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:15] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.048 second response time [13:57:07] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.023 second response time [13:57:13] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [13:57:17] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.016 second response time [13:57:39] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.017 second response time [13:57:40] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0004939138889 secs [13:57:53] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:11] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:12] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 11.04, 11.74 [13:59:06] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 1.027 second response time [13:59:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:14] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:59:18] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:32] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [13:59:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [14:00:16] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.061 second response time [14:00:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.65, 11.31, 11.75 [14:00:39] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:00:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.66, 3.47 [14:00:58] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:13] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [14:01:14] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.94, 10.90, 11.49 [14:01:23] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:01:27] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:12] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:28] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:30] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:32] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:44] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [14:02:48] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:03:04] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 37% [14:03:05] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.076 second response time [14:03:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 10.42, 11.26 [14:03:19] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:34] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:43] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11419 MB (29% inode=96%); [14:04:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 3.23, 3.37 [14:04:58] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.267 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:05:07] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:05:12] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:05:29] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:05:38] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.166 second response time [14:06:03] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.026 second response time [14:06:04] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:05] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:06:23] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.020 second response time [14:07:09] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.176 second response time [14:07:23] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.037 second response time [14:07:28] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:07:41] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0007697939873 secs [14:07:42] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:21] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:54] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:09:28] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.099 second response time [14:09:32] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [14:09:33] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:10:03] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 2.038 second response time [14:10:12] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:15] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.017 second response time [14:10:28] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:34] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.021 second response time [14:10:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 4.06, 3.62 [14:11:00] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.347 second response time [14:11:10] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.030 second response time [14:11:12] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [14:11:45] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:11:51] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:11:57] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:02] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:12] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:28] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:12:29] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0006913542747 secs [14:12:46] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:46] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.018 second response time [14:12:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.32, 3.39 [14:13:21] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:13:38] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 55% [14:13:47] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:13:48] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.048 second response time [14:13:56] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [14:14:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.06, 9.25, 10.12 [14:14:50] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.97, 8.94, 10.01 [14:15:20] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.176 second response time [14:15:42] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.118 second response time [14:16:02] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:16:22] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:16:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:16:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:16:52] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:16:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:59] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 0.043 second response time [14:17:04] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:17:06] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:07] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 2.068 second response time [14:17:11] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:17:18] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 57% [14:17:21] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:05] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:11] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:29] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.032 second response time [14:18:37] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:48] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [14:18:52] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.039 second response time [14:19:03] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [14:20:01] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.135 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:20:13] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:35] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.072 second response time [14:20:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.48, 3.29 [14:20:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:21:02] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:21:04] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:21:14] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.136 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:21:21] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.149 second response time [14:21:29] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [14:21:51] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [14:22:12] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 46% [14:22:14] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:22:14] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:22:19] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:37] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [14:22:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.51, 3.33 [14:23:07] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:23:19] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:23:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.46, 2.85 [14:23:48] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:14] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.047 second response time [14:24:18] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [14:24:24] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.040 second response time [14:24:30] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.060 second response time [14:24:37] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.861 second response time [14:24:58] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:25:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.90, 12.00, 10.70 [14:25:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 10.84, 10.43 [14:25:32] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.42, 2.91 [14:26:00] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.044 second response time [14:26:21] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.038 second response time [14:26:34] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:26:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.07, 3.20 [14:26:49] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:49] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:54] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:26:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.03, 0.01 [14:27:09] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.185 second response time [14:27:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.72, 11.21, 10.56 [14:27:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.034 second response time [14:27:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.10, 4.13, 3.21 [14:27:45] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.051 second response time [14:27:52] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:27:59] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.284 second response time [14:28:10] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.045 second response time [14:28:20] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 2.069 second response time [14:28:46] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.347 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:28:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 50% [14:29:04] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [14:29:10] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:29:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.74, 11.95, 10.89 [14:29:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.18, 11.99, 10.94 [14:29:19] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:36] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:05] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.061 second response time [14:30:19] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 0.041 second response time [14:30:33] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:30:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.45, 3.34 [14:31:07] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.11, 10.30, 10.44 [14:31:22] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.68, 3.25 [14:31:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.046 second response time [14:31:50] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [14:32:04] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11336 MB (29% inode=96%); [14:32:11] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:19] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.208 second response time [14:32:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 4.15, 3.63 [14:32:51] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:33:04] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001437753439 secs [14:33:28] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:30] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:47] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.071 second response time [14:33:54] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 54% [14:34:01] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:34:01] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:32] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.041 second response time [14:34:33] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:34:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.74, 3.53 [14:35:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.80, 11.69, 10.94 [14:35:16] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:21] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:35:35] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.058 second response time [14:36:14] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:16] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:17] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.187 second response time [14:36:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.046 second response time [14:36:26] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.050 second response time [14:36:29] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:36:50] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:37:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.76, 11.77, 11.31 [14:37:13] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:37:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.81, 10.09, 10.44 [14:37:15] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:37:34] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.176 second response time [14:38:09] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11410 MB (29% inode=96%); [14:38:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.09, 3.30 [14:38:43] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.073 second response time [14:38:45] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:39:01] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:33] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:40:45] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.029 second response time [14:40:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.28, 11.30, 11.18 [14:41:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 20.40, 13.82, 11.72 [14:41:31] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.13, 3.31 [14:41:43] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:46] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:42:10] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.136 second response time [14:42:22] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [14:42:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:42:29] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:42:52] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35755 bytes in 0.033 second response time [14:42:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 11.11, 11.14 [14:42:59] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:43:21] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:43:40] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:43:45] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:20] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [14:44:40] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [14:44:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.81, 12.29, 11.57 [14:45:13] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:45:34] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:45:35] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:45:38] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:45:43] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.150 second response time [14:45:53] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.042 second response time [14:45:57] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:03] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.184 second response time [14:46:14] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:46:23] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.025 second response time [14:46:25] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.90, 3.54 [14:46:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.93, 11.99, 11.57 [14:47:29] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.082 second response time [14:47:35] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:47:46] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:47:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:00] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:11] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.141 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:48:18] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 3.35, 3.39 [14:48:26] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.027 second response time [14:48:29] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [14:48:29] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [14:48:39] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.168 second response time [14:48:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.78, 12.13, 11.65 [14:48:53] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.531 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:49:56] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.053 second response time [14:50:05] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.022 second response time [14:50:26] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 47% [14:50:44] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [14:51:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.42, 10.94, 11.42 [14:51:28] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:51:32] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:51:52] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:51:58] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:52:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.43, 3.43 [14:52:32] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.26, 11.78, 11.72 [14:52:49] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:52:49] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:03] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:18] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:21] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:53:58] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.052 second response time [14:54:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.04, 3.28 [14:54:25] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.365 second response time [14:54:33] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.070 second response time [14:54:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.60, 11.99, 11.78 [14:54:44] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.056 second response time [14:55:01] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:55:46] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:56:24] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [14:56:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.019 second response time [14:56:33] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 28% [14:56:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.90, 10.70, 11.32 [14:57:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.62, 11.85, 11.48 [14:57:28] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:30] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:57:43] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.140 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [14:58:04] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:58:09] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11405 MB (29% inode=96%); [14:58:20] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:26] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:58:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.56, 12.17, 11.83 [14:58:56] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:38] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:59:38] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:59:43] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11332 MB (29% inode=96%); [14:59:53] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:01] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.150 second response time [15:00:22] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.71, 11.75, 11.72 [15:00:48] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.02, 0.07, 0.05 [15:01:30] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.103 second response time [15:01:36] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:01:43] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.042 second response time [15:02:00] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.026 second response time [15:02:25] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:02:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.49, 12.25, 11.91 [15:02:34] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:37] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:03] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.017 second response time [15:03:13] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:24] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.032 second response time [15:03:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [15:03:35] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.085 second response time [15:03:54] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [15:04:12] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:20] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:31] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0006939172745 secs [15:04:34] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:36] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:37] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:04:50] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:04:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.037 second response time [15:04:50] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [15:04:51] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:01] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 11.55, 11.72 [15:05:24] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:27] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.205 second response time [15:05:50] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.029 second response time [15:06:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.065 second response time [15:06:20] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:06:24] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.288 second response time [15:06:32] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 7.158 second response time [15:06:42] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.017 second response time [15:06:57] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:59] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:07:24] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.077 second response time [15:07:53] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 43% [15:08:36] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:41] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:08:45] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:59] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:00] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:09:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.43, 11.46, 11.57 [15:09:40] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.42, 10.34, 11.49 [15:10:26] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.045 second response time [15:10:59] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.039 second response time [15:11:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 11.13, 11.44 [15:11:14] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:11:25] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:11:25] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.469 second response time [15:11:51] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [15:11:59] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.054 second response time [15:12:19] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:12:20] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:12:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:12:55] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:12:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:13:02] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:13:14] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.368 second response time [15:13:17] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [15:13:18] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:13:36] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:41] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:13:43] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:15] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:22] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:15:23] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:15:30] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 7.239 second response time [15:15:43] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:50] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:15:52] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.182 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:16:13] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:16:20] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [15:16:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:17:13] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:21] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:17:27] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:34] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [15:17:49] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.034 second response time [15:17:51] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [15:18:34] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:18:51] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [15:19:11] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:15] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:19:15] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:19] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 7.290 second response time [15:19:30] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.021 second response time [15:19:34] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0003516972065 secs [15:19:47] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:19:52] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.061 second response time [15:19:54] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.017 second response time [15:21:06] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:21:13] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.201 second response time [15:21:31] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.026 second response time [15:21:31] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:40] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11399 MB (29% inode=96%); [15:21:58] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:22:15] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.049 second response time [15:22:19] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [15:22:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.37, 11.40, 10.89 [15:22:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 3.049 second response time [15:23:00] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.142 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:23:14] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.001418024302 secs [15:23:29] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:31] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11328 MB (29% inode=96%); [15:24:26] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.190 second response time [15:24:44] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:24:55] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 55% [15:25:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.47, 5.96, 4.19 [15:25:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.64, 11.51, 10.99 [15:25:35] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:25:39] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:25:48] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:26:26] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:26:52] [discord] I'm getting a 502 gateway when editing this page https://meta.miraheze.org/w/index.php?title=User:Bukkit/Sig&veaction=editsource [15:26:53] [url] User:Bukkit/Sig - Miraheze Meta | meta.miraheze.org [15:27:01] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.018 second response time [15:27:06] [discord] oh never mind [15:27:09] [discord] sorry [15:27:11] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:27:31] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:27:54] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:27:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:28:24] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.085 second response time [15:28:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.88, 3.05 [15:28:36] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 49% [15:28:49] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:55] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:29:07] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:29:09] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:29:12] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:29:12] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.052 second response time [15:29:15] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.023 second response time [15:29:27] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:29:43] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.035 second response time [15:29:48] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:29:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 44% [15:30:00] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.049 second response time [15:30:04] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:12] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.276 second response time [15:31:16] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:17] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.03, 0.00 [15:31:32] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:39] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.037 second response time [15:31:48] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:31:55] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:58] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.307 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [15:32:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.94, 3.30 [15:32:52] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.061 second response time [15:32:55] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:56] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 7 minutes ago with 0 failures [15:33:01] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.218 second response time [15:33:48] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [15:33:48] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.046 second response time [15:33:56] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:33:59] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:34:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.65, 4.09, 3.44 [15:34:19] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.155 second response time [15:34:28] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [15:34:32] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:34:34] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:34:56] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 37% [15:35:04] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.037 second response time [15:35:13] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0008355677128 secs [15:35:14] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.048 second response time [15:35:49] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:35:57] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:36:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 3.36, 3.25 [15:36:20] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.050 second response time [15:36:25] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 54% [15:36:43] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:45] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [15:36:54] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:37:45] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:38:03] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:38:15] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 7.270 second response time [15:38:27] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.028 second response time [15:38:36] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:37] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:38:56] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:38:59] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:15] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.368 second response time [15:39:28] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:28] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:46] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:54] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.031 second response time [15:40:01] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:40:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:19] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:24] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:24] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:41] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:53] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:40:56] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11324 MB (29% inode=96%); [15:40:56] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.163 second response time [15:41:01] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.031 second response time [15:41:19] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:41:39] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:42:10] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:19] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:22] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.048 second response time [15:42:42] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.042 second response time [15:42:48] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.052 second response time [15:42:54] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:42:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 58% [15:43:23] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:27] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:43:30] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:44:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 10.44, 11.93 [15:44:32] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [15:44:46] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 7.321 second response time [15:44:59] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.030 second response time [15:45:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.68, 3.54, 3.41 [15:45:16] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.01, 0.00 [15:45:29] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001389920712 secs [15:45:33] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.057 second response time [15:45:37] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.220 second response time [15:45:48] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 1.067 second response time [15:45:56] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:58] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:45:58] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:46:09] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 58% [15:46:10] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [15:46:53] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:46:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.30, 3.26, 3.33 [15:46:59] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:47:24] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.021 second response time [15:47:29] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:47:48] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [15:47:49] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [15:47:57] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:48:09] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:48:36] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:48:51] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:55] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:49:05] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:49:21] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [15:50:13] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:50:16] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:33] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:50:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.06, 3.75 [15:51:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 11.08, 11.93 [15:51:15] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 42% [15:51:21] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:51:23] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:51:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:09] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:13] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:52:14] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:20] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.056 second response time [15:52:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 11.35, 11.67 [15:52:36] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.058 second response time [15:52:42] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.034 second response time [15:53:00] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:53:19] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [15:53:23] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 48% [15:53:26] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 38% [15:53:32] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [15:53:51] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [15:53:56] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [15:54:08] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:54:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 10.69, 11.40 [15:54:37] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 0.033 second response time [15:54:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.58, 3.80 [15:55:33] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:55:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 4.17, 3.68 [15:56:16] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:19] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:22] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:34] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:53] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11390 MB (29% inode=96%); [15:56:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:57:27] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:57:36] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:57:41] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.017 second response time [15:58:09] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.016 second response time [15:58:09] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:12] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [15:58:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.66, 12.07, 11.72 [15:58:23] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 39% [15:58:25] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:48] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0001323223114 secs [15:58:51] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [15:59:34] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [16:00:05] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:00:07] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.044 second response time [16:00:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 11.90, 11.71 [16:00:30] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:32] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [16:00:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.83, 3.87 [16:00:49] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [16:01:17] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.045 second response time [16:01:19] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:33] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:10] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [16:02:19] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:22] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:02:29] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.020 second response time [16:03:02] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:03:15] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:16] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 58% [16:03:21] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:03:22] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:03:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.63, 3.73 [16:04:06] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:18] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:04:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.67, 11.32, 11.43 [16:04:55] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:05] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:20] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.66, 3.72 [16:05:46] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:05:52] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.188 second response time [16:05:54] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:01] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:06:11] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 10.35, 11.09 [16:06:52] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:13] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [16:07:19] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:07:23] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:07:27] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [16:07:27] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.021 second response time [16:07:28] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:07:49] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:03] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:08:20] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [16:08:30] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.056 second response time [16:08:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.86, 3.78 [16:08:49] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.138 second response time [16:08:50] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.017 second response time [16:08:50] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.063 second response time [16:09:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.00, 0.00, 0.00 [16:09:12] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.000767827034 secs [16:10:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.75, 3.76 [16:10:58] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.015 second response time [16:11:04] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [16:11:09] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [16:11:19] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [16:11:20] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.062 second response time [16:11:29] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.051 second response time [16:11:52] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:57] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.048 second response time [16:12:01] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.064 second response time [16:12:02] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [16:12:08] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:12:32] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:12:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.27, 4.30, 3.95 [16:12:43] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:13:03] [02puppet] 07Universal-Omega synchronize pull request 03#2780: README: non-profit -> not-for-profit - 13https://github.com/miraheze/puppet/pull/2780 [16:13:04] [url] Page not found · GitHub · GitHub | github.com [16:13:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 11.24, 11.01 [16:13:21] [02puppet] 07Universal-Omega edited pull request 03#2780: Update README - 13https://github.com/miraheze/puppet/pull/2780 [16:13:21] [url] Page not found · GitHub · GitHub | github.com [16:13:24] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.045 second response time [16:13:48] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:13:57] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:09] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:14:41] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:14:42] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.297 second response time [16:15:00] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:15:09] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 10.33, 10.70 [16:15:25] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:15:37] [02puppet] 07Universal-Omega edited pull request 03#2780: Update README - 13https://github.com/miraheze/puppet/pull/2780 [16:15:38] [url] Page not found · GitHub · GitHub | github.com [16:15:39] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.130 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:15:45] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:16:03] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3386 bytes in 3.106 second response time [16:16:31] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:16:36] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:05] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99038 bytes in 1.441 second response time [16:17:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.66, 9.03, 10.17 [16:17:18] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [16:17:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.91, 3.97 [16:17:34] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 46% [16:17:37] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:39] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:13] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 47% [16:18:29] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.018 second response time [16:18:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.62, 3.80 [16:18:47] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:19:07] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:22] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:19:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.10, 4.51, 4.16 [16:19:35] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.019 second response time [16:20:03] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.076 second response time [16:20:23] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:33] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:20:51] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [16:20:59] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:21:03] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:21:22] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:21:45] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:22:20] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:22:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:22:30] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:23:04] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [16:23:17] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:23:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.90, 3.98 [16:23:38] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [16:23:42] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:13] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 7.211 second response time [16:24:15] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:24:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.41, 10.71, 10.67 [16:24:24] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 1.031 second response time [16:24:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.60, 4.28, 3.98 [16:24:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.015 second response time [16:24:35] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.045 second response time [16:24:59] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:14] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.164 second response time [16:25:16] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:25:16] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 7.304 second response time [16:25:20] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:25] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:25:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.40, 4.63, 4.25 [16:25:52] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:03] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.16, 10.64, 10.63 [16:26:42] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [16:26:49] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:26:58] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:27:45] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:28:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.037 second response time [16:28:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.38, 3.70 [16:28:44] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:18] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:29:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.60, 3.93 [16:29:36] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:29:38] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.387 second response time [16:29:42] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.057 second response time [16:29:44] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.043 second response time [16:30:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.91, 9.30, 10.13 [16:30:23] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [16:30:28] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:30:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.75, 3.79 [16:30:32] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:30:44] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:02] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:02] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 52% [16:31:09] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:31:40] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.162 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:31:41] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:09] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:32:23] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.095 second response time [16:32:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.32, 3.62 [16:32:43] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3400 bytes in 0.015 second response time [16:33:07] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.156 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:33:21] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.220 second response time [16:33:40] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:47] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:48] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:01] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:02] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.036 second response time [16:34:12] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 1.031 second response time [16:34:18] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.060 second response time [16:35:03] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:53] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 3.032 second response time [16:36:01] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 45% [16:36:19] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [16:36:25] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.225 second response time [16:36:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.57, 3.25 [16:36:36] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [16:37:11] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:37:13] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.024 second response time [16:38:41] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [16:39:00] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:39:13] [02puppet] 07Universal-Omega opened pull request 03#2781: Convert LICENSE to markdown - 13https://github.com/miraheze/puppet/pull/2781 [16:39:13] [url] Page not found · GitHub · GitHub | github.com [16:39:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:39:22] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 2.75, 3.29 [16:39:39] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:00] [02puppet] 07Universal-Omega edited pull request 03#2781: Convert LICENSE to markdown - 13https://github.com/miraheze/puppet/pull/2781 [16:40:00] ... [16:40:47] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.08, 0.07, 0.03 [16:40:47] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.041 second response time [16:40:54] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:57] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:59] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 36% [16:41:13] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:41:20] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:41:23] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:41:25] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.027 second response time [16:41:46] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:06] [02puppet] 07Reception123 closed pull request 03#2780: Update README - 13https://github.com/miraheze/puppet/pull/2780 [16:42:06] [url] Page not found · GitHub · GitHub | github.com [16:42:07] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/caadb81ef61f...eabf937e4374 [16:42:08] [url] Comparing caadb81ef61f...eabf937e4374 · miraheze/puppet · GitHub | github.com [16:42:09] [02miraheze/puppet] 07Universal-Omega 03eabf937 - Update README (#2780) [16:42:23] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:27] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:30] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11315 MB (29% inode=96%); [16:43:04] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:10] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:11] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:43:21] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [16:43:22] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35770 bytes in 7.224 second response time [16:43:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.48, 3.43 [16:43:56] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.033 second response time [16:45:01] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3399 bytes in 7.121 second response time [16:45:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.54, 4.20, 3.79 [16:45:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.32, 3.38 [16:45:47] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:59] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:46:01] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.033 second response time [16:46:01] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:22] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.091 second response time [16:47:23] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.042 second response time [16:47:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.04, 4.44, 3.76 [16:47:35] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.060 second response time [16:47:44] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:47:45] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:47:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:21] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:43] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:03] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:04] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:23] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:24] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.62, 3.54 [16:49:39] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.394 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:49:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:49:44] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:49:50] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.275 second response time [16:50:04] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.011 second response time [16:50:10] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.060 second response time [16:50:12] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.053 second response time [16:50:21] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.187 second response time [16:50:51] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:59] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:51:02] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.039 second response time [16:51:03] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [16:51:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.044 second response time [16:51:06] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:51:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 10.05, 9.21 [16:51:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.60, 3.75 [16:51:23] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 11313 MB (29% inode=96%); [16:51:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.06, 0.02, 0.00 [16:52:00] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:13] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:52:41] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.001617610455 secs [16:52:56] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 48% [16:53:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.72, 9.97, 9.31 [16:53:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 4.40, 3.85 [16:53:33] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:37] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:53:38] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:54:01] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:17] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11374 MB (29% inode=96%); [16:54:34] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [16:55:13] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.108 second response time [16:55:20] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.066 second response time [16:55:20] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.017 second response time [16:55:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 0.019 second response time [16:55:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.93, 3.75 [16:55:55] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:16] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:17] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:32] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.150 second response time [16:56:33] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:41] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.083 second response time [16:56:41] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:56:43] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.035 second response time [16:56:44] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.141 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [16:56:54] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 0 % full [16:57:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 4.08, 3.88 [16:57:20] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 4.11, 3.84 [16:57:38] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.001026898623 secs [16:58:31] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [16:58:32] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.043 second response time [16:58:47] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [16:58:50] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [16:58:56] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:59:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.49, 11.49, 10.15 [16:59:21] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:59:31] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.254 second response time [16:59:34] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:59:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:06] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:15] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:54] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 53% [17:01:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.59, 11.45, 10.27 [17:01:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:01:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.86, 3.85 [17:01:47] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:49] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [17:01:54] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:02:04] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:02:31] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.037 second response time [17:02:50] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:01] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.075 second response time [17:03:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 10.39, 10.03 [17:03:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.66, 4.66, 4.14 [17:03:38] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:39] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:45] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:17] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.094 second response time [17:04:37] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset -0.0004629194736 secs [17:04:44] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:04:47] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.138 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:05:26] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:28] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:41] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:47] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:55] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 3.049 second response time [17:06:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.50, 11.85, 10.48 [17:07:05] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.034 second response time [17:07:08] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.91, 10.16, 10.06 [17:07:25] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.044 second response time [17:07:27] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [17:07:28] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:30] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.048 second response time [17:08:04] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:12] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:17] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:08:52] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.025 second response time [17:08:55] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/d06a5449c19e...1419824744fe [17:08:56] [url] Comparing d06a5449c19e...1419824744fe · miraheze/landing · GitHub | github.com [17:08:57] [02miraheze/landing] 07Universal-Omega 031419824 - Disable site notice [17:09:03] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:09:03] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:09:19] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.017 second response time [17:09:31] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.024 second response time [17:09:35] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 59% [17:09:53] miraheze/landing - Universal-Omega the build passed. [17:10:07] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:15] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:21] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.38, 10.83, 10.31 [17:10:24] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:10:59] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 11369 MB (29% inode=96%); [17:11:30] PROBLEM - cp21 NTP time on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:11:35] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:12:18] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0004876554012 secs [17:12:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 10.71, 10.36 [17:12:29] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 5.092 second response time [17:12:30] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:12:30] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 58% [17:12:32] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 2.045 second response time [17:13:12] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.054 second response time [17:13:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.07, 10.85, 10.35 [17:13:22] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.019 second response time [17:13:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.62, 3.93 [17:13:33] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 6118 bytes in 7.069 second response time [17:13:59] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:14:04] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:14:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 9.82, 10.08 [17:14:37] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 1.031 second response time [17:15:00] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.015 second response time [17:15:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.01, 0.02, 0.00 [17:15:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.14, 4.08 [17:15:57] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14728 bytes in 1.019 second response time [17:16:02] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14738 bytes in 0.023 second response time [17:16:24] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.133 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [17:16:35] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [17:16:47] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 35724 bytes in 1.038 second response time [17:16:50] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.047 second response time [17:17:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.71, 3.92 [17:17:36] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:17:44] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:06] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 35762 bytes in 0.057 second response time [17:18:16] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.011 second response time [17:18:19] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.55, 3.94 [17:19:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.83, 10.16, 10.20 [17:19:26] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:30] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:31] RECOVERY - cp21 NTP time on cp21 is OK: NTP OK: Offset 0.0005379319191 secs [17:19:40] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:45] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 0.014 second response time [17:20:14] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [17:20:28] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:21:05] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:21:53] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:05] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.087 second response time [17:22:06] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:25] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 0 % full [17:22:25] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.081 second response time [17:22:27] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:22:37] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 59% [17:22:38] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:41] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 7.196 second response time [17:22:54] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:23:01] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14732 bytes in 7.215 second response time [17:23:21] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:30] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14736 bytes in 7.254 second response time [17:23:38] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:23:46] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:23:55] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:07] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14724 bytes in 3.050 second response time [17:28:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.20, 3.53 [17:30:44] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/dd67aa5dbc9f [17:30:45] [02miraheze/ssl] 07Universal-Omega 03dd67aa5 - Add wiki.artmechanicum.com cert [17:30:47] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:30:47] [url] Page not found · GitHub · GitHub | github.com [17:30:48] [02ssl] 07Universal-Omega opened pull request 03#586: Add wiki.artmechanicum.com cert - 13https://github.com/miraheze/ssl/pull/586 [17:30:49] ... [17:30:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.59, 3.66 [17:31:25] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dd67aa5dbc9f...ef290daac6b3 [17:31:26] [url] Comparing dd67aa5dbc9f...ef290daac6b3 · miraheze/ssl · GitHub | github.com [17:31:26] [02miraheze/ssl] 07Universal-Omega 03ef290da - Update certs.yaml [17:31:28] [02ssl] 07Universal-Omega synchronize pull request 03#586: Add wiki.artmechanicum.com cert - 13https://github.com/miraheze/ssl/pull/586 [17:31:28] [url] Page not found · GitHub · GitHub | github.com [17:31:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 2.93, 3.40 [17:33:11] [02ssl] 07Universal-Omega closed pull request 03#586: Add wiki.artmechanicum.com cert - 13https://github.com/miraheze/ssl/pull/586 [17:33:11] [url] Page not found · GitHub · GitHub | github.com [17:33:12] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:33:14] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:33:14] [url] Page not found · GitHub · GitHub | github.com [17:33:15] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/902a26ca9e92...aa0130534692 [17:33:16] [url] Comparing 902a26ca9e92...aa0130534692 · miraheze/ssl · GitHub | github.com [17:33:17] [02miraheze/ssl] 07Universal-Omega 03aa01305 - Add wiki.artmechanicum.com cert (#586) [17:34:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.53, 3.60 [17:35:48] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/b40d9530b314 [17:35:49] [02miraheze/ssl] 07Universal-Omega 03b40d953 - Add www.sekaipedia.org cert [17:35:51] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:35:51] [url] Page not found · GitHub · GitHub | github.com [17:35:52] [02ssl] 07Universal-Omega opened pull request 03#587: Add www.sekaipedia.org cert - 13https://github.com/miraheze/ssl/pull/587 [17:35:53] ... [17:36:34] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b40d9530b314...dd8b626c0f51 [17:36:35] [url] Comparing b40d9530b314...dd8b626c0f51 · miraheze/ssl · GitHub | github.com [17:36:36] [02miraheze/ssl] 07Universal-Omega 03dd8b626 - Update certs.yaml [17:36:37] [02ssl] 07Universal-Omega synchronize pull request 03#587: Add www.sekaipedia.org cert - 13https://github.com/miraheze/ssl/pull/587 [17:36:38] [url] Page not found · GitHub · GitHub | github.com [17:36:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.99, 3.39 [17:37:38] [02ssl] 07Universal-Omega closed pull request 03#587: Add www.sekaipedia.org cert - 13https://github.com/miraheze/ssl/pull/587 [17:37:39] ... [17:37:40] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/aa0130534692...ab694c40ea6a [17:37:40] [url] Comparing aa0130534692...ab694c40ea6a · miraheze/ssl · GitHub | github.com [17:37:41] [02miraheze/ssl] 07Universal-Omega 03ab694c4 - Add www.sekaipedia.org cert (#587) [17:37:43] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:37:44] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:37:45] [url] Page not found · GitHub · GitHub | github.com [17:39:35] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=nychousingwikiwiki /srv/mediawiki/w/extensions/Moderation/sql/patch-moderation-mod_type.sql [17:39:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:43:49] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/9f3efa7a3057 [17:43:51] [02miraheze/ssl] 07Universal-Omega 039f3efa7 - Add jakkidverse.pog.gay cert [17:43:52] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:43:52] [url] Page not found · GitHub · GitHub | github.com [17:43:54] [02ssl] 07Universal-Omega opened pull request 03#588: Add jakkidverse.pog.gay cert - 13https://github.com/miraheze/ssl/pull/588 [17:43:54] [url] Page not found · GitHub · GitHub | github.com [17:44:23] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9f3efa7a3057...3c609ece3646 [17:44:23] [url] Comparing 9f3efa7a3057...3c609ece3646 · miraheze/ssl · GitHub | github.com [17:44:24] [02miraheze/ssl] 07Universal-Omega 033c609ec - Update certs.yaml [17:44:26] [02ssl] 07Universal-Omega synchronize pull request 03#588: Add jakkidverse.pog.gay cert - 13https://github.com/miraheze/ssl/pull/588 [17:44:26] [url] Page not found · GitHub · GitHub | github.com [17:45:45] [02ssl] 07Universal-Omega closed pull request 03#588: Add jakkidverse.pog.gay cert - 13https://github.com/miraheze/ssl/pull/588 [17:45:46] ... [17:45:47] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:45:48] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:45:49] [url] Page not found · GitHub · GitHub | github.com [17:45:50] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/ab694c40ea6a...daa6dcc2dd62 [17:45:50] [url] Comparing ab694c40ea6a...daa6dcc2dd62 · miraheze/ssl · GitHub | github.com [17:45:51] [02miraheze/ssl] 07Universal-Omega 03daa6dcc - Add jakkidverse.pog.gay cert (#588) [17:48:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.48, 3.35 [17:50:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.90, 4.19, 3.61 [17:51:35] [02dns] 07Universal-Omega opened pull request 03#330: Remove usaha.click zone - 13https://github.com/miraheze/dns/pull/330 [17:51:35] [url] Page not found · GitHub · GitHub | github.com [17:53:00] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/739f73ce86a7 [17:53:01] [02miraheze/ssl] 07Universal-Omega 03739f73c - Delete www.wikitipp.de.crt [17:53:03] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [17:53:03] ... [17:53:19] [02ssl] 07Universal-Omega opened pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [17:53:20] [url] Page not found · GitHub · GitHub | github.com [17:53:54] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/739f73ce86a7...371277a20c7c [17:53:55] [url] Comparing 739f73ce86a7...371277a20c7c · miraheze/ssl · GitHub | github.com [17:53:56] [02miraheze/ssl] 07Universal-Omega 03371277a - Delete usaha.click.crt [17:53:57] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [17:53:58] [url] Page not found · GitHub · GitHub | github.com [17:54:13] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/371277a20c7c...4d6718df0d1c [17:54:13] [url] Comparing 371277a20c7c...4d6718df0d1c · miraheze/ssl · GitHub | github.com [17:54:14] [02miraheze/ssl] 07Universal-Omega 034d6718d - Delete dnd.bellinrattin.it.crt [17:54:16] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [17:54:16] [url] Page not found · GitHub · GitHub | github.com [17:54:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.10, 3.29 [17:57:04] [02dns] 07Universal-Omega synchronize pull request 03#330: Remove usaha.click zone - 13https://github.com/miraheze/dns/pull/330 [17:57:04] [url] Page not found · GitHub · GitHub | github.com [17:57:23] [02dns] 07Universal-Omega edited pull request 03#330: Remove no longer pointing custom domain zones - 13https://github.com/miraheze/dns/pull/330 [17:57:24] ... [17:58:10] [02dns] 07Universal-Omega synchronize pull request 03#330: Remove no longer pointing custom domain zones - 13https://github.com/miraheze/dns/pull/330 [17:58:10] [url] Page not found · GitHub · GitHub | github.com [17:58:39] [02dns] 07Universal-Omega synchronize pull request 03#330: Remove no longer pointing custom domain zones - 13https://github.com/miraheze/dns/pull/330 [17:58:39] ... [17:59:15] [02dns] 07Universal-Omega synchronize pull request 03#330: Remove no longer pointing custom domain zones - 13https://github.com/miraheze/dns/pull/330 [17:59:15] [url] Page not found · GitHub · GitHub | github.com [18:00:09] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/4d6718df0d1c...9f2250e99717 [18:00:10] [url] Comparing 4d6718df0d1c...9f2250e99717 · miraheze/ssl · GitHub | github.com [18:00:11] [02miraheze/ssl] 07Universal-Omega 039f2250e - Delete wiki.dereric.gq.crt [18:00:12] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:00:13] [url] Page not found · GitHub · GitHub | github.com [18:00:38] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/9f2250e99717...6aeb956dd6f2 [18:00:38] [url] Comparing 9f2250e99717...6aeb956dd6f2 · miraheze/ssl · GitHub | github.com [18:00:39] [02miraheze/ssl] 07Universal-Omega 036aeb956 - Delete wiki.wallymer.com.crt [18:00:41] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:00:41] [url] Page not found · GitHub · GitHub | github.com [18:01:14] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/6aeb956dd6f2...617989311004 [18:01:15] [url] Comparing 6aeb956dd6f2...617989311004 · miraheze/ssl · GitHub | github.com [18:01:16] [02miraheze/ssl] 07Universal-Omega 036179893 - Delete wiki.fraterniterebellion.com.crt [18:01:17] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:01:18] [url] Page not found · GitHub · GitHub | github.com [18:02:15] [02dns] 07Universal-Omega synchronize pull request 03#330: Remove no longer pointing custom domain zones - 13https://github.com/miraheze/dns/pull/330 [18:02:15] [url] Page not found · GitHub · GitHub | github.com [18:02:26] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/617989311004...d2b180cd0e87 [18:02:27] [url] Comparing 617989311004...d2b180cd0e87 · miraheze/ssl · GitHub | github.com [18:02:28] [02miraheze/ssl] 07Universal-Omega 03d2b180c - Delete docse.tk.crt [18:02:29] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:02:30] [url] Page not found · GitHub · GitHub | github.com [18:02:46] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/d2b180cd0e87...748bb86ede0e [18:02:46] [url] Comparing d2b180cd0e87...748bb86ede0e · miraheze/ssl · GitHub | github.com [18:02:47] [02miraheze/ssl] 07Universal-Omega 03748bb86 - Delete sbs.wiki.crt [18:02:49] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:02:49] [url] Page not found · GitHub · GitHub | github.com [18:03:10] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/748bb86ede0e...e2a301004580 [18:03:11] [url] Comparing 748bb86ede0e...e2a301004580 · miraheze/ssl · GitHub | github.com [18:03:12] [02miraheze/ssl] 07Universal-Omega 03e2a3010 - Delete otcg.ml.crt [18:03:13] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:03:14] [url] Page not found · GitHub · GitHub | github.com [18:03:21] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/e2a301004580...ec8cbfbfb916 [18:03:22] [url] Comparing e2a301004580...ec8cbfbfb916 · miraheze/ssl · GitHub | github.com [18:03:23] [02miraheze/ssl] 07Universal-Omega 03ec8cbfb - Delete en.pornwiki.org.crt [18:03:24] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:03:25] [url] Page not found · GitHub · GitHub | github.com [18:03:31] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/ec8cbfbfb916...0fedbe77ec95 [18:03:31] [url] Comparing ec8cbfbfb916...0fedbe77ec95 · miraheze/ssl · GitHub | github.com [18:03:32] [02miraheze/ssl] 07Universal-Omega 030fedbe7 - Delete files.pornwiki.org.crt [18:03:34] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:03:34] [url] Page not found · GitHub · GitHub | github.com [18:03:39] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/0fedbe77ec95...cdb4e8ce53ae [18:03:40] [url] Comparing 0fedbe77ec95...cdb4e8ce53ae · miraheze/ssl · GitHub | github.com [18:03:41] [02miraheze/ssl] 07Universal-Omega 03cdb4e8c - Delete pornwiki.org.crt [18:03:42] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:03:43] [url] Page not found · GitHub · GitHub | github.com [18:03:49] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/cdb4e8ce53ae...7e50242cd1a0 [18:03:49] [url] Comparing cdb4e8ce53ae...7e50242cd1a0 · miraheze/ssl · GitHub | github.com [18:03:50] [02miraheze/ssl] 07Universal-Omega 037e50242 - Delete www.pornwiki.org.crt [18:03:52] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:03:52] [url] Page not found · GitHub · GitHub | github.com [18:04:11] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/7e50242cd1a0...3ee44730efed [18:04:12] [url] Comparing 7e50242cd1a0...3ee44730efed · miraheze/ssl · GitHub | github.com [18:04:13] [02miraheze/ssl] 07Universal-Omega 033ee4473 - Delete rww2.org.crt [18:04:14] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:04:15] [url] Page not found · GitHub · GitHub | github.com [18:06:34] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3ee44730efed...57e5ff35f3fa [18:06:35] [url] Comparing 3ee44730efed...57e5ff35f3fa · miraheze/ssl · GitHub | github.com [18:06:36] [02miraheze/ssl] 07Universal-Omega 0357e5ff3 - Update redirects.yaml [18:06:37] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:06:38] [url] Page not found · GitHub · GitHub | github.com [18:08:38] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/57e5ff35f3fa...d09fe47a935e [18:08:38] [url] Comparing 57e5ff35f3fa...d09fe47a935e · miraheze/ssl · GitHub | github.com [18:08:39] [02miraheze/ssl] 07Universal-Omega 03d09fe47 - Update certs.yaml [18:08:41] [02ssl] 07Universal-Omega synchronize pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:08:41] [url] Page not found · GitHub · GitHub | github.com [18:14:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.87, 10.31, 8.45 [18:15:32] PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'webkinzguide.com' expires in 15 day(s) (Mon 29 Aug 2022 18:06:45 GMT +0000). [18:16:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.62, 9.70, 8.45 [18:22:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/daa6dcc2dd62...d1bf3e48d9d7 [18:22:26] [url] Comparing daa6dcc2dd62...d1bf3e48d9d7 · miraheze/ssl · GitHub | github.com [18:22:27] [02miraheze/ssl] 07MirahezeSSLBot 03d1bf3e4 - Bot: Update SSL cert for webkinzguide.com [18:27:45] paladox, Reception123: can https://github.com/miraheze/puppet/pull/2699 be merged? [18:27:46] [url] ssl: extend revoke option to also remove key from git by MacFan4000 · Pull Request #2699 · miraheze/puppet · GitHub | github.com [18:28:50] [02puppet] 07paladox closed pull request 03#2699: ssl: extend revoke option to also remove key from git - 13https://github.com/miraheze/puppet/pull/2699 [18:28:50] [url] Page not found · GitHub · GitHub | github.com [18:28:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eabf937e4374...425d197f7497 [18:28:52] [url] Comparing eabf937e4374...425d197f7497 · miraheze/puppet · GitHub | github.com [18:28:53] [02miraheze/puppet] 07MacFan4000 03425d197 - ssl: extend revoke option to also remove key from git (#2699) [18:28:57] Thanks! [18:52:04] [02ssl] 07Universal-Omega closed pull request 03#589: Remove no longer pointing custom domains - 13https://github.com/miraheze/ssl/pull/589 [18:52:05] [url] Page not found · GitHub · GitHub | github.com [18:52:06] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-14/±2] 13https://github.com/miraheze/ssl/compare/d1bf3e48d9d7...6c5392f4de45 [18:52:06] [url] Comparing d1bf3e48d9d7...6c5392f4de45 · miraheze/ssl · GitHub | github.com [18:52:07] [02miraheze/ssl] 07Universal-Omega 036c5392f - Remove no longer pointing custom domains (#589) [18:52:09] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:52:10] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [18:52:11] [url] Page not found · GitHub · GitHub | github.com [18:53:39] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.fraterniterebellion.com.key],File[/etc/ssl/private/en.pornwiki.org.key],File[/etc/ssl/private/files.pornwiki.org.key],File[/etc/ssl/private/usaha.click.key] [18:53:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 10.38, 9.28 [18:55:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.51, 11.59, 10.05 [18:56:49] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 3 minutes ago with 14 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.fraterniterebellion.com.key],File[/etc/ssl/private/en.pornwiki.org.key],File[/etc/ssl/private/files.pornwiki.org.key],File[/etc/ssl/private/usaha.click.key] [18:57:32] [02dns] 07Reception123 closed pull request 03#330: Remove no longer pointing custom domain zones - 13https://github.com/miraheze/dns/pull/330 [18:57:32] [url] Page not found · GitHub · GitHub | github.com [18:57:33] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-6/±0] 13https://github.com/miraheze/dns/compare/11793c0bf3b5...ea2b65e94af0 [18:57:34] [url] Comparing 11793c0bf3b5...ea2b65e94af0 · miraheze/dns · GitHub | github.com [18:57:35] [02miraheze/dns] 07Universal-Omega 03ea2b65e - Remove no longer pointing custom domain zones (#330) [18:59:27] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki.fraterniterebellion.com.key],File[/etc/ssl/private/en.pornwiki.org.key],File[/etc/ssl/private/files.pornwiki.org.key],File[/etc/ssl/private/usaha.click.key] [19:01:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 11.66, 10.70 [19:03:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.62, 11.29, 10.21 [19:05:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 10.67, 10.11 [19:07:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.33, 9.48, 9.75 [19:13:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.97, 8.41, 9.61 [19:17:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.25, 11.08, 10.38 [19:18:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 10.32, 10.00 [19:20:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.44, 13.05, 11.01 [19:21:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.00, 11.02, 10.56 [19:22:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 11.68, 10.75 [19:24:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.40, 9.77, 10.14 [19:24:27] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:24:42] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.32, 2.66 [19:27:19] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:29:51] [02dns] 07Universal-Omega opened pull request 03#331: Remove wiki-asterix.cf zone - 13https://github.com/miraheze/dns/pull/331 [19:29:52] [url] Page not found · GitHub · GitHub | github.com [19:30:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.75, 3.02 [19:30:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 4.08, 3.25 [19:30:29] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/7ff526ad1e44 [19:30:30] ... [19:30:31] [02miraheze/ssl] 07Universal-Omega 037ff526a - Remove wiki-asterix.cf cert [19:30:32] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:30:33] [url] Page not found · GitHub · GitHub | github.com [19:30:36] [02ssl] 07Universal-Omega opened pull request 03#590: Remove wiki-asterix.cf cert - 13https://github.com/miraheze/ssl/pull/590 [19:30:37] ... [19:30:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.44, 10.71, 10.41 [19:31:08] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7ff526ad1e44...9ee90741b4c6 [19:31:08] [url] Comparing 7ff526ad1e44...9ee90741b4c6 · miraheze/ssl · GitHub | github.com [19:31:09] [02miraheze/ssl] 07Universal-Omega 039ee9074 - Update certs.yaml [19:31:11] [02ssl] 07Universal-Omega synchronize pull request 03#590: Remove wiki-asterix.cf cert - 13https://github.com/miraheze/ssl/pull/590 [19:31:11] [url] Page not found · GitHub · GitHub | github.com [19:32:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 4.30, 3.31 [19:32:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 10.34, 10.33 [19:34:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.98, 3.32 [19:34:19] [02ssl] 07Universal-Omega closed pull request 03#590: Remove wiki-asterix.cf cert - 13https://github.com/miraheze/ssl/pull/590 [19:34:20] [url] Page not found · GitHub · GitHub | github.com [19:34:21] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/6c5392f4de45...b1a695723b3b [19:34:22] [url] Comparing 6c5392f4de45...b1a695723b3b · miraheze/ssl · GitHub | github.com [19:34:22] [02miraheze/ssl] 07Universal-Omega 03b1a6957 - Remove wiki-asterix.cf cert (#590) [19:34:24] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [19:34:24] [url] Page not found · GitHub · GitHub | github.com [19:34:25] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:36:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.08, 9.05, 9.88 [19:37:08] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki-asterix.cf.key] [19:38:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.89, 3.58 [19:38:27] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki-asterix.cf.key] [19:38:34] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki-asterix.cf.key] [19:39:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.09, 3.90, 3.44 [19:41:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 21.52, 14.04, 11.07 [19:41:51] RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Fri 11 Nov 2022 17:22:21 GMT +0000. [19:42:41] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki-asterix.cf.key] [19:43:18] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wiki-asterix.cf.key] [19:44:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.22, 5.01, 4.00 [19:44:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.15, 9.98, 9.84 [19:46:36] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b1a695723b3b...5ad4d092b816 [19:46:37] [url] Comparing b1a695723b3b...5ad4d092b816 · miraheze/ssl · GitHub | github.com [19:46:38] [02miraheze/ssl] 07Universal-Omega 035ad4d09 - Add projectsekaiwiki redirect [19:46:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.92, 10.16, 9.94 [19:47:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.87, 11.16, 10.78 [19:51:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 9.48, 10.17 [19:52:28] !log [universalomega@test131] starting deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-1', 'config': True} to all [19:52:31] !log [universalomega@test131] finished deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-1', 'config': True} to all - SUCCESS in 2s [19:52:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.61, 3.87 [19:55:02] [02mw-config] 07Universal-Omega closed pull request 03#4864: MirahezeFunctions: Convert DB select queries to use SelectQueryBuilder - 13https://github.com/miraheze/mw-config/pull/4864 [19:55:03] [url] Page not found · GitHub · GitHub | github.com [19:55:04] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:55:04] [url] Page not found · GitHub · GitHub | github.com [19:55:05] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:55:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e8096627c8ee...f8708e3a222d [19:55:07] [url] Comparing e8096627c8ee...f8708e3a222d · miraheze/mw-config · GitHub | github.com [19:55:08] [02miraheze/mw-config] 07Universal-Omega 03f8708e3 - MirahezeFunctions: Convert DB select queries to use SelectQueryBuilder (#4864) [19:56:06] !log [universalomega@test131] starting deploy of {'pull': 'config', 'branch': 'master', 'config': True} to all [19:56:07] !log [universalomega@test131] finished deploy of {'pull': 'config', 'branch': 'master', 'config': True} to all - SUCCESS in 1s [19:56:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:17] miraheze/mw-config - Universal-Omega the build passed. [19:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.91, 12.99, 11.40 [19:56:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 4.52, 4.21 [19:56:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.62, 12.65, 11.04 [20:00:07] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [20:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:00:21] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [20:00:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:23] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:06:33] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:06:54] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:07:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.97, 11.37, 11.75 [20:09:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 19.73, 13.55, 12.42 [20:10:01] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:10:36] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:21:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.22, 3.90 [20:23:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.50, 3.91 [20:24:34] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:33:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.62, 3.90 [20:39:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.73, 3.83 [20:41:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.86, 3.87 [20:43:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 3.99, 3.89 [20:45:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 10.41, 11.88 [20:47:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.98, 3.92 [20:50:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.03, 3.77 [20:52:34] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:53:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.54, 2.55, 3.29 [20:56:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 2.54, 3.30 [20:56:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f8708e3a222d...476b31efff25 [20:56:42] [url] Comparing f8708e3a222d...476b31efff25 · miraheze/mw-config · GitHub | github.com [20:56:43] [02miraheze/mw-config] 07Universal-Omega 03476b31e - Disable editnotify again [20:57:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.51, 8.67, 10.13 [20:57:43] miraheze/mw-config - Universal-Omega the build passed. [20:59:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.80, 11.98 [21:01:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/mw-config/commit/d8b7e36f0c07 [21:01:32] [02miraheze/mw-config] 07Universal-Omega 03d8b7e36 - Convert LICENSE to markdown [21:01:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:01:34] [url] Page not found · GitHub · GitHub | github.com [21:01:41] [02mw-config] 07Universal-Omega opened pull request 03#4871: Convert LICENSE to markdown - 13https://github.com/miraheze/mw-config/pull/4871 [21:01:41] [url] Page not found · GitHub · GitHub | github.com [21:02:48] miraheze/mw-config - Universal-Omega the build passed. [21:03:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.92, 13.25, 12.52 [21:04:25] [02mw-config] 07Universal-Omega closed pull request 03#4871: Convert LICENSE to markdown - 13https://github.com/miraheze/mw-config/pull/4871 [21:04:26] [url] Page not found · GitHub · GitHub | github.com [21:04:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/mw-config/compare/476b31efff25...badfb8c27a8c [21:04:27] [url] Comparing 476b31efff25...badfb8c27a8c · miraheze/mw-config · GitHub | github.com [21:04:28] [02miraheze/mw-config] 07Universal-Omega 03badfb8c - Convert LICENSE to markdown (#4871) [21:04:30] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:04:31] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:04:32] ... [21:05:31] miraheze/mw-config - Universal-Omega the build passed. [21:07:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 3.20, 2.98 [21:07:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.94, 10.15, 11.47 [21:09:32] !log [@mwtask141] starting deploy of {'config': True} to all [21:09:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:41] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [21:09:43] [02mw-config] 07Universal-Omega closed pull request 03#4827: add moderation email to managewiki settings - 13https://github.com/miraheze/mw-config/pull/4827 [21:09:44] [url] Page not found · GitHub · GitHub | github.com [21:09:45] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Reception123-patch-1 [21:09:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/badfb8c27a8c...f16c6492832d [21:09:47] [url] Comparing badfb8c27a8c...f16c6492832d · miraheze/mw-config · GitHub | github.com [21:09:48] [02miraheze/mw-config] 07Reception123 03f16c649 - add moderation email to managewiki settings (#4827) [21:09:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 2.93, 2.90 [21:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:49] [02mw-config] 07Universal-Omega deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [21:09:50] [url] Page not found · GitHub · GitHub | github.com [21:10:53] miraheze/mw-config - Universal-Omega the build passed. [21:11:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.86, 11.44, 11.62 [21:13:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.45, 3.45 [21:13:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:15:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.48, 5.18, 4.08 [21:15:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.93, 11.81, 10.38 [21:17:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 11.89, 11.92 [21:17:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.38, 10.84, 10.19 [21:17:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [21:19:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.78, 3.72 [21:19:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.63, 10.14, 10.02 [21:23:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.18, 3.80, 3.70 [21:25:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.32, 3.46 [21:26:34] !log [@test131] starting deploy of {'config': True} to all [21:26:35] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:26:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.96, 3.43 [21:29:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.74, 8.37, 9.92 [21:31:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.76, 3.06, 3.20 [21:35:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.62, 3.31, 3.92 [21:37:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.35, 9.76, 9.30 [21:39:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.92, 11.19, 9.85 [21:39:42] !log [@mwtask141] starting deploy of {'config': True} to all [21:39:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:51] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [21:40:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.90, 11.27, 10.56 [21:43:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 11.56, 10.45 [21:45:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.65, 12.50, 10.92 [21:47:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 2.65, 3.33 [21:47:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:51:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.89, 11.75, 11.06 [21:51:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [21:55:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.66, 6.00, 4.40 [21:55:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 12.20, 6.98, 4.50 [21:55:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:03:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.84, 3.79, 3.95 [22:06:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [22:07:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 4.02, 4.01 [22:09:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5590a9c0c199 [22:09:38] [02miraheze/mw-config] 07Universal-Omega 035590a9c - Update LocalSettings.php [22:09:40] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [22:09:40] [url] Page not found · GitHub · GitHub | github.com [22:09:54] [02mw-config] 07Universal-Omega opened pull request 03#4872: T9645: add `$wgDisplayTitleFollowRedirects` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4872 [22:09:54] ... [22:10:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:11:02] miraheze/mw-config - Universal-Omega the build passed. [22:14:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5590a9c0c199...74a1e9c04641 [22:14:08] [url] Comparing 5590a9c0c199...74a1e9c04641 · miraheze/mw-config · GitHub | github.com [22:14:09] [02miraheze/mw-config] 07Universal-Omega 0374a1e9c - Update ManageWikiSettings.php [22:14:10] [02mw-config] 07Universal-Omega synchronize pull request 03#4872: T9645: add `$wgDisplayTitleFollowRedirects` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4872 [22:14:11] [url] Page not found · GitHub · GitHub | github.com [22:15:13] miraheze/mw-config - Universal-Omega the build passed. [22:15:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.57, 3.94 [22:16:58] [02mw-config] 07Universal-Omega closed pull request 03#4872: T9645: add `$wgDisplayTitleFollowRedirects` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4872 [22:16:58] [url] Page not found · GitHub · GitHub | github.com [22:16:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/f16c6492832d...fa1d06328202 [22:17:00] [url] Comparing f16c6492832d...fa1d06328202 · miraheze/mw-config · GitHub | github.com [22:17:01] [02miraheze/mw-config] 07Universal-Omega 03fa1d063 - T9645: add `$wgDisplayTitleFollowRedirects` to ManageWikiSettings (#4872) [22:17:02] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [22:17:04] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [22:17:04] [url] Page not found · GitHub · GitHub | github.com [22:17:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 4.00, 4.05 [22:18:13] miraheze/mw-config - Universal-Omega the build passed. [22:19:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.89, 3.99 [22:21:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 4.01, 4.00 [22:23:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.60, 3.91 [22:23:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.21, 9.96, 11.50 [22:25:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.70, 11.37, 11.80 [22:26:16] !log [@test131] starting deploy of {'config': True} to all [22:26:17] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [22:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:26:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:27:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.43, 3.74 [22:31:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.77, 5.06, 4.30 [22:33:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 2.73, 3.32 [22:35:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.48, 3.82 [22:39:46] !log [@mwtask141] starting deploy of {'config': True} to all [22:39:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:39:55] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [22:40:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:41:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.07, 3.47 [22:45:22] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4214 MB (10% inode=96%); [22:45:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.57, 3.66 [22:47:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 10.57, 11.85 [22:49:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 2.83, 3.32 [22:59:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.78, 8.81, 10.08 [23:01:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.91, 9.75, 11.58 [23:02:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.38, 1.75, 1.96 [23:03:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.45, 11.63, 12.01 [23:04:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 1.91, 1.99 [23:05:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.36, 11.50, 11.96 [23:06:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.95, 2.00 [23:08:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.76, 2.30, 2.12 [23:11:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.85, 10.28, 11.11 [23:13:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.03, 10.38, 11.06 [23:16:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 2.89, 2.62 [23:17:14] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4222 MB (10% inode=96%); [23:18:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.27, 2.78 [23:20:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 2.85, 2.68 [23:27:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.23, 10.74, 10.79 [23:30:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.51, 1.89 [23:30:44] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 16.56, 15.77, 9.79 [23:30:45] PROBLEM - mw132 nutcracker process on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [23:33:44] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 25035 bytes in 0.204 second response time [23:34:44] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 6.55, 10.35, 8.91 [23:36:44] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.43, 8.69, 8.47 [23:37:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.67, 11.99, 11.93 [23:38:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.86, 1.89 [23:39:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 12.15, 12.03 [23:40:17] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.73, 1.42, 1.72 [23:40:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 2.93, 2.72 [23:41:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.21, 2.84 [23:41:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.51, 11.35, 10.05 [23:42:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.00, 1.75, 1.79 [23:42:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.43, 2.42, 2.57 [23:43:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 10.51, 9.89 [23:43:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.34, 3.14, 2.86 [23:43:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 11.50, 11.86 [23:45:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.85, 8.87, 9.36 [23:51:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.68, 7.74, 9.95 [23:59:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.72, 10.18, 9.43 [23:59:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.90, 15.22, 12.26