[00:00:30] miraheze/mw-config - paladox the build passed. [00:00:40] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:00:40] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:33] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [00:03:10] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:18] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:19] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:21] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:22] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:23] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:30] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:34] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:36] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:40] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:45] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:53] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:53] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:57] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [00:03:57] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:00] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:02] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:14] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:16] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:21] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [00:04:25] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:27] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:27] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:31] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:33] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 51% [00:04:35] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:35] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:39] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [00:04:46] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:01] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:05:02] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:03] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:07] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:10] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:16] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [00:05:21] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.321 second response time [00:05:21] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:05:53] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.030 second response time [00:05:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:05:57] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.056 second response time [00:06:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [00:06:21] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 23% [00:06:26] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.048 second response time [00:06:30] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [00:06:48] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [00:07:44] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 7.212 second response time [00:07:44] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.617 second response time [00:07:44] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 1.322 second response time [00:07:44] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [00:07:46] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.007 second response time [00:07:53] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 4.816 second response time [00:08:01] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.007 second response time [00:08:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 8.855 second response time [00:08:04] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.016 second response time [00:08:09] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [00:08:10] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [00:08:21] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [00:08:24] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [00:08:25] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.019 second response time [00:08:27] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [00:08:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.371 second response time [00:08:31] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 10% [00:08:35] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.307 second response time [00:08:40] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [00:08:45] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.314 second response time [00:08:47] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [00:08:48] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 19% [00:08:55] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.334 second response time [00:09:03] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.179 second response time [00:09:05] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [00:09:12] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [00:09:19] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.331 second response time [00:09:24] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [00:09:31] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [00:09:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [00:09:35] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.333 second response time [00:09:56] !log [@mwtask141] starting deploy of {'config': True} to all [00:09:57] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [00:09:57] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw121.miraheze.org [00:10:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:53] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 23.25, 11.89, 7.04 [00:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:08] paladox: you may need to manually deploy, since servers are down. [00:11:19] ok [00:11:22] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [00:11:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 2.75, 5.44, 11.35 [00:11:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 2.48, 3.91 [00:11:55] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [00:11:55] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [00:11:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:03] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 8s [00:12:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:09] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [00:12:14] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [00:12:53] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.40, 10.64, 7.23 [00:12:59] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [00:13:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 3.51, 4.13 [00:14:52] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.48, 8.73, 6.94 [00:17:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.83, 9.85, 11.21 [00:18:10] RECOVERY - db112 NTP time on db112 is OK: NTP OK: Offset -0.009786278009 secs [00:18:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.06, 1.38, 0.99 [00:19:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.29, 10.44, 11.28 [00:22:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.72, 1.35 [00:22:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.22, 1.47, 1.12 [00:23:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.99, 12.54, 11.88 [00:23:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/632227763b9e [00:23:42] [02miraheze/puppet] 07paladox 036322277 - gluster: adjust vm.dirty_ratio vm.dirty_background_ratio [00:23:44] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [00:23:44] [url] Page not found · GitHub · GitHub | github.com [00:23:45] [02puppet] 07paladox opened pull request 03#2764: gluster: adjust vm.dirty_ratio vm.dirty_background_ratio - 13https://github.com/miraheze/puppet/pull/2764 [00:23:46] ... [00:23:58] !log deploy https://github.com/miraheze/puppet/pull/2764 to gluster* as a experiment [00:23:58] [url] gluster: adjust vm.dirty_ratio vm.dirty_background_ratio by paladox · Pull Request #2764 · miraheze/puppet · GitHub | github.com [00:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.49, 1.60, 1.35 [00:26:34] !log [@test131] starting deploy of {'config': True} to all [00:26:35] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [00:26:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.47, 9.43, 7.86 [00:30:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.05, 8.70, 7.77 [00:32:35] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [00:38:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.16, 10.63, 9.07 [00:40:08] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:42:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.41, 10.17, 9.26 [00:51:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.58, 2.65, 3.86 [00:52:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.68, 1.98, 1.54 [00:56:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.08, 1.76, 1.57 [00:57:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 13.17, 6.70, 5.01 [00:57:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.73, 1.44 [00:58:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.77, 2.88, 1.93 [00:59:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.47, 1.62, 1.43 [01:01:39] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [01:10:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.99, 1.65, 1.99 [01:16:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 1.62, 1.85 [01:20:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.86, 1.91 [01:22:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.12, 2.14, 1.99 [01:28:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.74, 1.88 [01:28:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.461 second response time [01:32:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 2.13, 1.98 [01:34:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.34, 2.93, 2.08 [01:38:05] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.94, 3.82, 3.04 [01:38:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.75, 1.91 [01:40:26] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.83, 1.84 [01:41:54] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.03, 2.97, 2.87 [01:42:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.62, 1.39, 1.67 [01:42:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.52, 1.16, 1.64 [01:45:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.83, 3.94 [01:49:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.35, 5.16, 4.43 [01:53:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.58, 3.68, 3.98 [01:57:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.81, 10.75, 9.51 [01:57:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.82, 5.23, 4.43 [01:59:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.97, 11.26, 9.88 [02:04:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.66, 11.46, 10.37 [02:06:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.75, 9.78, 9.89 [02:09:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.54, 3.98 [02:09:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.39, 4.18, 3.46 [02:10:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 11.37, 10.54 [02:11:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.97, 3.84, 3.44 [02:11:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.98, 2.61, 2.02 [02:12:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 11.35, 10.67 [02:13:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.39, 3.86, 3.48 [02:15:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.70, 3.85 [02:19:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.08, 3.88, 3.98 [02:20:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.97, 9.21, 9.94 [02:21:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.67, 4.10, 4.06 [02:25:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.64, 3.92 [02:27:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.41, 4.63, 4.20 [02:29:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.75, 3.55, 3.87 [02:29:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.73, 11.91, 10.86 [02:29:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.16, 3.96, 3.94 [02:29:54] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:54] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:02] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [02:30:38] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [02:30:56] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:59] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.94, 11.41, 10.78 [02:31:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.22, 4.14, 4.02 [02:31:55] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.025 second response time [02:32:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.20, 1.65, 1.95 [02:32:53] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.059 second response time [02:33:00] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.353 second response time [02:33:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.59, 12.20, 11.17 [02:33:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.43, 3.78 [02:33:45] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 5.928 second response time [02:33:57] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.461 second response time [02:34:13] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.108 second response time [02:34:45] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.333 second response time [02:34:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.81, 1.90, 2.02 [02:35:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.28, 11.81, 11.16 [02:35:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.24, 3.77, 3.84 [02:37:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 2.51, 3.29 [02:37:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.06, 3.53, 3.74 [02:40:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.51, 12.16, 11.28 [02:42:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.48, 11.77, 11.25 [02:45:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.19, 2.50, 3.25 [02:46:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.74, 3.56 [02:46:38] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:38] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:40] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.78, 1.56, 1.96 [02:46:57] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:08] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.015 second response time [02:47:31] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:46] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:48] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:49] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.78, 11.82, 11.27 [02:48:37] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.345 second response time [02:48:38] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.350 second response time [02:48:39] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.378 second response time [02:48:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 4 backends are down. mw121 mw122 mw131 mw141 [02:48:53] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [02:49:07] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.352 second response time [02:49:16] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.313 second response time [02:49:31] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.343 second response time [02:49:35] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [02:49:42] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [02:49:46] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.308 second response time [02:49:47] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.318 second response time [02:50:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.70, 4.09, 3.68 [02:50:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.51, 8.63, 10.16 [02:50:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 1.82, 1.97 [02:51:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 1.89, 1.59 [02:52:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [02:53:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.73, 1.42, 1.45 [02:53:36] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [02:54:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.83, 1.93 [02:56:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.32, 9.27, 9.78 [02:57:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.50, 3.73, 3.40 [02:58:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.91, 9.15, 9.68 [02:58:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.68, 1.18, 1.64 [03:01:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.20, 3.89, 3.53 [03:03:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.73, 3.85, 3.56 [03:05:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.44, 4.12, 3.70 [03:06:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.92, 2.07, 1.87 [03:07:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.16, 10.76, 10.18 [03:07:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.79, 3.86, 3.64 [03:08:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.92, 1.84 [03:09:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.08, 3.97, 3.68 [03:11:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.35, 3.93, 3.72 [03:12:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.51, 1.70 [03:12:55] [discord] Wiki request queue seems to be completely broken: https://meta.miraheze.org/wiki/Special:RequestWikiQueue/26041 [03:12:55] [url] Wiki requests queue - Miraheze Meta | meta.miraheze.org [03:13:04] [discord] I can't get a single request to load [03:13:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.06, 11.42, 10.64 [03:13:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.77, 5.08, 4.16 [03:14:12] CosmicAlpha: ^ [03:15:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.14, 10.47, 10.41 [03:15:16] I have not deployed anything at all last few days (to CW anyway). Could it be the db cluster change today paladox? [03:15:40] Can someone check Graylog for error? [03:15:41] @Agent what error do you get [03:15:52] [discord] fa923510487790acbec66e8a [03:15:52] I see the exception. [03:16:06] [discord] I can't sign in to Graylog because of the 503s lol [03:16:36] Agent: the Graylog issue is not 503 related. [03:16:43] Graylog is not behind cp* [03:16:54] So not behind varnish at all. [03:16:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.35, 9.15, 9.93 [03:17:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.03, 3.59, 3.77 [03:17:49] I could login to graylog yesterday but Reception said they could not either. [03:17:51] [discord] CosmicAlpha: Yeah though it's strange that Graylog tells me it couldn't sign be in because "the server returned: 503" [03:22:45] paladox: https://community.graylog.org/t/after-graylog-2-5-to-3-0-upgrade-getting-503-service-unavailable/13379 could that be the Graylog issue? I actually just tried logging back into Graylog myself also, no longer works either, strangely did yesterday though. [03:22:46] [url] After graylog 2.5 to 3.0 upgrade getting 503 Service Unavailable - Graylog Central - Graylog Community | community.graylog.org [03:22:54] it's ldap [03:22:58] i just figured it out [03:23:02] but i cannot login lol [03:23:06] i have to use the root thingy [03:23:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.98, 2.62, 3.29 [03:25:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.91, 2.00, 1.76 [03:27:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.78, 1.71 [03:28:20] @Agent CosmicAlpha Fixed [03:29:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.38, 1.56 [03:31:23] https://graylog.miraheze.org/messages/graylog_5/8b443540-146c-11ed-bc48-5a209e913d91 [03:31:24] huh [03:32:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.52, 1.80, 1.54 [03:34:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.48, 1.68, 1.52 [03:35:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 2.07, 1.79 [03:36:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.89, 1.76 [03:38:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.57, 1.65 [03:42:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 2.06, 1.83 [03:44:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.81, 1.77 [03:46:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 1.85, 1.78 [03:48:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.65, 1.71 [03:52:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.74, 1.72 [03:54:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.40, 1.59 [04:11:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.56, 2.45, 3.83 [04:15:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.74, 1.55 [04:17:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 3.21, 3.65 [04:17:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.53, 1.49 [04:19:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.51, 3.71 [04:22:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.75, 1.51 [04:23:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 10.63, 11.77 [04:24:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.53, 1.45 [04:25:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.00, 3.75 [04:25:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.29, 11.50, 11.95 [04:41:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.94, 11.05, 9.69 [04:43:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.15, 9.26, 9.17 [04:59:33] [discord] Thanks! [05:01:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.02, 3.55, 2.82 [05:03:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.32, 3.16, 2.77 [05:07:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.31, 4.23, 3.29 [05:09:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.04, 3.75, 3.23 [05:13:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.76, 11.05, 9.78 [05:15:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 10.39, 9.71 [05:15:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.02, 4.11, 3.51 [05:17:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.86, 9.44, 9.46 [05:21:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 13.56, 6.27, 2.95 [05:23:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.12, 3.75, 3.63 [05:25:49] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.70, 4.22, 3.80 [05:26:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 10.76, 10.02 [05:29:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.58, 1.73, 1.99 [05:32:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.73, 10.05, 10.02 [05:35:20] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.56, 1.02, 1.58 [05:36:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.16, 10.68, 10.22 [05:37:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.08, 3.33, 3.97 [05:38:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 10.59, 10.27 [05:39:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.50, 3.93 [05:40:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.05, 11.70, 10.72 [05:41:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.98, 3.08, 3.74 [05:42:48] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.03, 3.63, 3.84 [05:43:53] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 25035 bytes in 0.853 second response time [05:44:36] PROBLEM - mw131 nutcracker process on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [05:44:47] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [05:45:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.591 second response time [05:45:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 7.75, 12.30, 8.80 [05:46:06] PROBLEM - mw132 nutcracker process on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [05:46:33] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.15, 15.33, 9.42 [05:46:37] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 25035 bytes in 0.281 second response time [05:47:07] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.49, 10.58, 8.58 [05:47:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.32, 3.25, 3.97 [05:47:50] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 8.12, 5.20, 3.05 [05:48:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [05:48:27] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [05:49:07] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.06, 8.56, 8.08 [05:50:32] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 5.41, 9.86, 8.51 [05:51:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 0.89, 1.90, 3.23 [05:52:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:52:27] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:54:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 11.75, 11.89 [05:55:53] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [05:58:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:59:13] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:01:14] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:12] !log restart nutcracker on mw* [06:03:12] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.016 second response time [06:03:37] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.019 second response time [06:03:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw141 [06:06:02] !log [paladox@mwtask141] starting deploy of {'l10n': True} to all [06:06:34] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw122.miraheze.org [06:06:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.55, 11.36, 11.29 [06:06:53] !log [paladox@mwtask141] starting deploy of {'l10n': True, 'force': True} to all [06:06:57] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw122 mw142 [06:07:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4844-paladox-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/e1a431d8590c [06:07:27] [02miraheze/mw-config] 07Universal-Omega 03e1a431d - Revert "Add db131/db141 (#4844)" [06:07:29] [02mw-config] 07Universal-Omega created branch 03revert-4844-paladox-patch-2 - 13https://github.com/miraheze/mw-config [06:07:29] [url] Page not found · GitHub · GitHub | github.com [06:07:31] [02mw-config] 07Universal-Omega opened pull request 03#4845: Revert "Add db131/db141" - 13https://github.com/miraheze/mw-config/pull/4845 [06:07:31] [url] Page not found · GitHub · GitHub | github.com [06:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:42] [02mw-config] 07Universal-Omega closed pull request 03#4845: Revert "Add db131/db141" - 13https://github.com/miraheze/mw-config/pull/4845 [06:07:42] [url] Page not found · GitHub · GitHub | github.com [06:07:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/bdedc9eef8a6...b964a269a3fa [06:07:44] [url] Comparing bdedc9eef8a6...b964a269a3fa · miraheze/mw-config · GitHub | github.com [06:07:45] [02miraheze/mw-config] 07Universal-Omega 03b964a26 - Revert "Add db131/db141" (#4845) [06:07:47] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4844-paladox-patch-2 [06:07:48] [02mw-config] 07Universal-Omega deleted branch 03revert-4844-paladox-patch-2 - 13https://github.com/miraheze/mw-config [06:07:48] [url] Page not found · GitHub · GitHub | github.com [06:08:05] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [06:08:17] !log [paladox@mwtask141] finished deploy of {'l10n': True, 'force': True} to all - SUCCESS in 83s [06:08:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:08:32] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [06:08:33] miraheze/mw-config - Universal-Omega the build passed. [06:08:51] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw131.miraheze.org [06:09:00] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [06:09:04] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.04, 1.54, 1.21 [06:09:07] miraheze/mw-config - Universal-Omega the build passed. [06:09:15] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.46, 3.97 [06:09:21] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 21s [06:09:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:09:56] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [06:10:13] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0) [06:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:10:56] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:58] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.54, 1.52, 1.24 [06:11:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 3.82, 4.03 [06:11:23] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.039 second response time [06:12:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/64b317913555 [06:12:45] [02miraheze/mw-config] 07Universal-Omega 0364b3179 - Fix use of undefined constants [06:12:47] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [06:12:47] [url] Page not found · GitHub · GitHub | github.com [06:12:48] [02mw-config] 07Universal-Omega opened pull request 03#4846: Fix use of undefined constants - 13https://github.com/miraheze/mw-config/pull/4846 [06:12:49] ... [06:13:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.72, 3.98 [06:13:43] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:56] miraheze/mw-config - Universal-Omega the build passed. [06:14:35] !log rebooting mw131 - nutcracker failing with it stating that something is already listening on the port??? [06:14:36] RECOVERY - mw131 nutcracker process on mw131 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [06:15:05] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:15:11] [02mw-config] 07Universal-Omega closed pull request 03#4846: Fix use of undefined constants - 13https://github.com/miraheze/mw-config/pull/4846 [06:15:12] [url] Page not found · GitHub · GitHub | github.com [06:15:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b964a269a3fa...63a9a3c245ca [06:15:14] [url] Comparing b964a269a3fa...63a9a3c245ca · miraheze/mw-config · GitHub | github.com [06:15:15] [02miraheze/mw-config] 07Universal-Omega 0363a9a3c - Fix use of undefined constants (#4846) [06:15:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:15:17] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [06:15:18] ... [06:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 4.04, 4.06 [06:15:37] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.364 second response time [06:16:22] miraheze/mw-config - Universal-Omega the build passed. [06:16:36] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:16:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.24, 10.49, 11.84 [06:16:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.12, 1.84, 1.47 [06:18:37] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:18:42] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.87, 1.82, 1.50 [06:19:16] !log rebooting mw132 - nutcracker failing with it stating that something is already listening on the port??? [06:20:06] RECOVERY - mw132 nutcracker process on mw132 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [06:20:37] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.226 second response time [06:20:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.51, 2.09, 1.64 [06:21:09] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw122 mw141 [06:21:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 11.38, 11.95 [06:21:50] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 1.79, 4.28, 5.89 [06:22:42] PROBLEM - wiki.infomedia.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:20] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw142 [06:24:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 10.26, 10.83 [06:24:38] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:07] PROBLEM - source.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.01, 11.20, 11.64 [06:25:50] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.09, 2.58, 4.83 [06:26:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.86, 9.81, 10.59 [06:26:38] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:26:44] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:06] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:23] PROBLEM - roblox-wiki.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:16] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [06:28:28] !log [@test131] starting deploy of {'config': True} to all [06:28:29] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:28:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:28:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:57] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.156 second response time [06:30:59] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:32:59] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:34:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.49, 8.36, 9.80 [06:35:15] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0) [06:35:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:55] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.48, 8.92, 11.39 [06:39:58] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Tue 23 Aug 2022 01:40:20 GMT +0000. [06:40:10] !log [@mwtask141] starting deploy of {'config': True} to all [06:40:20] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [06:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:25] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.034 second response time [06:40:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:42:17] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:42:48] RECOVERY - wikileague.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wikileague.cf' will expire on Mon 22 Aug 2022 22:21:46 GMT +0000. [06:42:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.98, 2.06, 1.40 [06:44:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.65, 1.33 [06:45:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.42, 2.59, 3.78 [06:47:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.40, 7.68, 9.92 [06:48:25] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:30] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:52] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw131 mw132 [06:49:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.10, 2.02, 3.29 [06:50:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:50:28] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:50:59] PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:42] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.63, 1.84, 1.98 [06:53:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.40, 12.55, 11.23 [06:54:08] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [06:54:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.34, 10.09, 10.49 [06:56:14] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [06:56:27] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.52, 1.96, 1.98 [06:56:52] RECOVERY - roblox-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'roblox-wiki.tk' will expire on Thu 27 Oct 2022 15:10:45 GMT +0000. [06:56:54] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:57:57] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:49] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.61, 3.40 [06:59:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.87, 8.78, 9.89 [06:59:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:00:42] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 0.76, 1.60, 1.85 [07:00:59] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 16.32, 10.34, 6.60 [07:01:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.40, 3.40, 3.34 [07:02:59] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.26, 10.01, 6.99 [07:03:02] PROBLEM - theapplesecretswiki.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:03:10] PROBLEM - reviwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:04:42] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 0.35, 0.98, 1.54 [07:05:17] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:05:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.50, 3.38 [07:05:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 10.46, 10.30 [07:05:35] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:44] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:07:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.39, 8.96, 9.75 [07:08:44] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.025 second response time [07:09:01] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:09:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 3.93, 3.57 [07:10:04] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.292 second response time [07:10:06] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:17] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:40] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw121 mw131 mw141 [07:11:08] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.053 second response time [07:11:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.64, 3.52 [07:11:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.37, 3.88 [07:12:03] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:12:43] PROBLEM - thestories.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.59, 2.95, 3.28 [07:13:24] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:02] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [07:15:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.23, 9.22, 9.32 [07:16:35] PROBLEM - lcn.zfc.id.lv - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.58, 8.57, 9.09 [07:19:40] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:19:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.58, 3.69 [07:19:59] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:21:09] PROBLEM - rarewiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:31] RECOVERY - wiki.infomedia.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Fri 21 Oct 2022 15:33:32 GMT +0000. [07:21:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.62, 3.70 [07:21:54] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.030 second response time [07:21:56] PROBLEM - jp.popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:22:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:24:05] PROBLEM - taswin.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:25:02] PROBLEM - istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:25:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.65, 10.32, 8.43 [07:26:23] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [07:26:29] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [07:27:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.67, 11.37, 9.04 [07:28:06] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Thu 01 Sep 2022 00:43:35 GMT +0000. [07:29:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.79, 10.15, 8.86 [07:29:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.98, 8.71, 8.10 [07:29:38] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.87, 4.32, 3.85 [07:31:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.26, 9.18, 8.39 [07:31:43] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.315 second response time [07:32:24] RECOVERY - theapplesecretswiki.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'theapplesecretswiki.ml' will expire on Mon 17 Oct 2022 19:49:07 GMT +0000. [07:32:59] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:48] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.95, 3.82 [07:34:26] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Mon 26 Sep 2022 13:36:29 GMT +0000. [07:34:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:43] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [07:36:57] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:37:32] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.064 second response time [07:38:48] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [07:38:52] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.312 second response time [07:40:43] PROBLEM - www.ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 4.12, 3.89 [07:41:49] PROBLEM - piwik.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:29] RECOVERY - thestories.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'thestories.kr' will expire on Thu 01 Sep 2022 00:44:22 GMT +0000. [07:42:33] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:22] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Wed 24 Aug 2022 04:03:58 GMT +0000. [07:43:23] PROBLEM - wiki.harlonserver.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:44:15] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:44:32] PROBLEM - miraheze.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:15] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:45] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.58, 3.77 [07:46:16] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.223 second response time [07:46:42] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Wed 31 Aug 2022 20:33:05 GMT +0000. [07:47:43] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [07:47:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.33, 4.83, 4.21 [07:48:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 31.12, 17.98, 12.07 [07:48:52] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:56] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:05] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:16] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Tue 23 Aug 2022 00:50:24 GMT +0000. [07:49:23] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:24] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:25] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 43.50, 22.95, 13.75 [07:49:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:08] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:15] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:30] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:50:36] PROBLEM - tr.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:43] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:46] PROBLEM - cloud12 Current Load on cloud12 is CRITICAL: CRITICAL - load average: 113.74, 77.63, 44.36 [07:51:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.72, 1.27 [07:51:25] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.199 second response time [07:51:27] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.039 second response time [07:51:57] PROBLEM - www.ethene.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:32] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.030 second response time [07:52:39] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:13] PROBLEM - wiki.infomedia.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.42, 1.21 [07:53:35] RECOVERY - istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [07:53:41] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:43] PROBLEM - smashbros.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.357 second response time [07:54:18] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.342 second response time [07:55:41] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [07:55:48] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [07:56:04] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.040 second response time [07:56:46] PROBLEM - cloud12 Current Load on cloud12 is WARNING: WARNING - load average: 17.66, 46.61, 42.67 [07:57:06] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:01] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:19] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.171 second response time [07:58:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.49, 10.31, 12.00 [07:58:46] RECOVERY - cloud12 Current Load on cloud12 is OK: OK - load average: 22.28, 39.20, 40.45 [07:58:47] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:42] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.087 second response time [08:02:23] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:30] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:45] PROBLEM - entw.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:02] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:16] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sat 17 Sep 2022 19:46:36 GMT +0000. [08:03:39] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:53] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:04:06] PROBLEM - hi.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:04:24] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:04:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.96, 5.85, 9.50 [08:04:59] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:09] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.040 second response time [08:05:38] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.034 second response time [08:05:44] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:05:49] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:05:51] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.029 second response time [08:05:54] PROBLEM - wiki.alathramc.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:21] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Wed 24 Aug 2022 03:45:41 GMT +0000. [08:06:23] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:07:37] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:07:44] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [08:08:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [08:08:49] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.059 second response time [08:08:59] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.48, 2.94 [08:11:48] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:12:01] RECOVERY - wiki.harlonserver.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.harlonserver.net' will expire on Wed 24 Aug 2022 16:27:40 GMT +0000. [08:12:12] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [08:12:37] PROBLEM - fortis.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:02] PROBLEM - test2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.24, 2.92 [08:14:24] RECOVERY - miraheze.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.tk' will expire on Tue 23 Aug 2022 07:48:08 GMT +0000. [08:14:35] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [08:14:51] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:11] PROBLEM - lgbtqia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:20] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:16:41] RECOVERY - lcn.zfc.id.lv - LetsEncrypt on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Wed 31 Aug 2022 21:26:51 GMT +0000. [08:16:50] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:17:49] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Sat 01 Oct 2022 16:09:21 GMT +0000. [08:17:53] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:00] PROBLEM - polandball.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:16] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [08:18:26] PROBLEM - donkeykong.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:31] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [08:18:36] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [08:18:52] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.060 second response time [08:20:06] RECOVERY - tr.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'tr.wikiretet.ga' will expire on Mon 12 Sep 2022 14:49:12 GMT +0000. [08:20:13] RECOVERY - rarewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewiki.org' will expire on Wed 28 Sep 2022 05:06:19 GMT +0000. [08:20:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.08, 1.44, 0.99 [08:21:02] RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Sat 24 Sep 2022 13:58:45 GMT +0000. [08:21:08] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Mon 10 Oct 2022 06:13:03 GMT +0000. [08:21:50] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:21:54] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:21:57] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:21:57] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:22:05] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:22:14] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:29] RECOVERY - wiki.infomedia.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Fri 21 Oct 2022 15:33:32 GMT +0000. [08:22:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.66, 1.22 [08:22:48] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.18, 0.94 [08:23:17] RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:23:51] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.025 second response time [08:24:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.53, 1.21 [08:25:18] PROBLEM - wikipariksha.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:25:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.85, 8.52, 11.31 [08:26:15] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [08:26:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.43, 1.22 [08:26:53] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [08:27:28] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:27:42] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [08:27:59] RECOVERY - arcanewaterswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'arcanewaterswiki.org' will expire on Sun 30 Oct 2022 20:54:06 GMT +0000. [08:28:55] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:29:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.50, 11.10, 11.58 [08:29:29] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:30:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.45, 4.08, 3.44 [08:30:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=eskomisanwiki --no-updates --username-prefix=wikia:komisan /mnt/mediawiki-static/metawiki/ImportDump/eskomisanwiki-20220801142035.xml (START) [08:30:33] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:37] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 1.039 second response time [08:30:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:31:09] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.076 second response time [08:31:14] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=eskomisanwiki --no-updates --username-prefix=wikia:komisan /mnt/mediawiki-static/metawiki/ImportDump/eskomisanwiki-20220801142035.xml (END - exit=0) [08:31:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=eskomisanwiki (START) [08:31:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.99, 11.00, 11.49 [08:31:31] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sat 17 Sep 2022 20:22:48 GMT +0000. [08:31:39] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:31:50] PROBLEM - experimyco.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:01] PROBLEM - atnarsia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:02] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.020 second response time [08:32:06] RECOVERY - entw.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'entw.ml' will expire on Tue 01 Nov 2022 04:38:34 GMT +0000. [08:32:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.87, 3.44 [08:32:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:32:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 11.17, 9.46 [08:33:04] RECOVERY - hi.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:44:45 GMT +0000. [08:33:16] PROBLEM - cp20 PowerDNS Recursor on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:33:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=eskomisanwiki (END - exit=0) [08:33:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=eskomisanwiki --active --update (END - exit=0) [08:33:54] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Tue 23 Aug 2022 01:47:24 GMT +0000. [08:34:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:34:10] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.021 second response time [08:34:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:34:36] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Mon 26 Sep 2022 13:36:29 GMT +0000. [08:34:43] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:14] RECOVERY - cp20 PowerDNS Recursor on cp20 is OK: DNS OK: 0.188 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [08:35:19] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [08:35:46] PROBLEM - metroid.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:57] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:22] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:25] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.65, 9.74, 9.37 [08:36:57] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Mon 10 Oct 2022 04:52:56 GMT +0000. [08:36:58] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.603 second response time [08:37:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.30, 7.71, 9.92 [08:37:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.69, 3.48 [08:37:57] PROBLEM - wiki.parappa.dev - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:01] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.049 second response time [08:38:21] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.240 second response time [08:38:27] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:29] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:38:52] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.072 second response time [08:38:57] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:29] PROBLEM - wiki.ameristraliagov.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:39] RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:39:44] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:39:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.82, 3.23, 3.33 [08:40:13] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:17] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:40:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:40:28] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Sat 17 Sep 2022 21:51:04 GMT +0000. [08:40:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.58, 10.95, 9.91 [08:40:56] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:20] PROBLEM - test1.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:31] PROBLEM - wiki.evscope.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:12] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:24] RECOVERY - fortis.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:42:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.33, 10.38, 9.82 [08:43:20] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:43:30] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [08:43:32] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:36] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:37] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:55] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:00] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:02] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:08] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:20] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:28] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:44:35] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.20, 9.23, 9.45 [08:44:48] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:49] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:55] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:05] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [08:45:05] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [08:45:09] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:45:18] PROBLEM - en.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:30] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.345 second response time [08:45:34] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:35] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:45:44] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:51] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [08:45:53] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.231 second response time [08:45:57] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.366 second response time [08:45:58] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 5.860 second response time [08:46:00] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.058 second response time [08:46:03] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:03] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:06] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [08:46:08] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [08:46:16] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:19] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.322 second response time [08:46:34] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.044 second response time [08:46:37] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.078 second response time [08:46:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [08:46:49] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.021 second response time [08:47:00] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw121 mw131 [08:47:05] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [08:47:05] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.319 second response time [08:47:06] RECOVERY - polandball.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:47:24] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:36] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:41] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.033 second response time [08:47:48] RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:48:06] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 3 % full [08:48:12] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.151 second response time [08:48:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:48:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw131 [08:49:08] RECOVERY - lg.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lg.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:37:50 GMT +0000. [08:49:27] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:37] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [08:49:51] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.044 second response time [08:50:46] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [08:51:00] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [08:51:17] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:22] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:28] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:29] PROBLEM - itora.bunnies1230.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:00] PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:01] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [08:52:08] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:22] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.927 second response time [08:53:13] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:17] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:53:24] PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 23.27, 14.39, 10.59 [08:53:35] PROBLEM - nintendo.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:41] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.126 second response time [08:54:09] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Wed 24 Aug 2022 05:18:49 GMT +0000. [08:54:57] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:17] PROBLEM - smashbros.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:43] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [08:55:57] PROBLEM - wiki.puucraft.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:30] PROBLEM - mario.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:50] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [08:56:51] PROBLEM - www.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:04] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:57:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 11.80, 10.19 [08:57:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.78, 11.20, 10.25 [08:58:00] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:53] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:13] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.00, 9.96, 9.74 [08:59:17] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:59:21] PROBLEM - animatedfeet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 12.02, 10.73 [08:59:39] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:00:01] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.219 second response time [09:00:26] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [09:01:18] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.054 second response time [09:01:37] RECOVERY - atnarsia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'atnarsia.com' will expire on Sun 25 Sep 2022 20:45:25 GMT +0000. [09:01:50] RECOVERY - experimyco.com - LetsEncrypt on sslhost is OK: OK - Certificate 'experimyco.com' will expire on Thu 15 Sep 2022 05:36:04 GMT +0000. [09:02:21] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:03:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [09:04:03] RECOVERY - wiki.alathramc.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.alathramc.com' will expire on Tue 20 Sep 2022 12:13:47 GMT +0000. [09:04:22] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.045 second response time [09:05:40] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:17] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:06:57] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [09:06:58] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [09:07:32] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:07:39] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.033 second response time [09:07:49] RECOVERY - www.ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Mon 22 Aug 2022 22:13:35 GMT +0000. [09:07:55] RECOVERY - wiki.parappa.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.parappa.dev' will expire on Sun 18 Sep 2022 13:44:45 GMT +0000. [09:07:59] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Wed 26 Oct 2022 20:54:04 GMT +0000. [09:08:18] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [09:08:20] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.026 second response time [09:09:03] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Wed 24 Aug 2022 03:08:19 GMT +0000. [09:09:07] RECOVERY - wiki.ameristraliagov.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ameristraliagov.com' will expire on Thu 20 Oct 2022 03:45:43 GMT +0000. [09:09:58] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [09:10:07] RECOVERY - wiki.evscope.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Fri 07 Oct 2022 11:21:52 GMT +0000. [09:10:16] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:26] RECOVERY - test1.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:10:35] RECOVERY - test2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:10:48] PROBLEM - phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:29] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Mon 05 Sep 2022 00:02:52 GMT +0000. [09:12:13] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:12:14] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [09:12:42] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:12:45] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Mon 22 Aug 2022 23:01:19 GMT +0000. [09:12:46] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:58:00 GMT +0000. [09:13:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.14, 9.91, 11.73 [09:13:30] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:42] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:13:49] RECOVERY - lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 07 Oct 2022 11:35:52 GMT +0000. [09:14:37] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:38] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [09:14:42] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [09:14:43] RECOVERY - en.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiretet.ga' will expire on Sun 02 Oct 2022 05:37:01 GMT +0000. [09:14:44] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [09:14:50] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [09:15:38] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.062 second response time [09:16:37] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:17:24] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:17:45] [02mw-config] 07ugochimobi opened pull request 03#4847: T9629: configure templateditor and extendedconfirmed rights for theredpionnerwiki - 13https://github.com/miraheze/mw-config/pull/4847 [09:17:45] [url] Page not found · GitHub · GitHub | github.com [09:18:13] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:18:27] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Sat 01 Oct 2022 16:09:21 GMT +0000. [09:18:31] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [09:18:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.45, 9.61, 8.20 [09:18:51] miraheze/mw-config - ugochimobi the build passed. [09:19:23] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:15] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [09:20:27] RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Sat 01 Oct 2022 16:22:56 GMT +0000. [09:20:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.13, 8.51, 7.94 [09:20:49] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:51] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Tue 01 Nov 2022 02:48:17 GMT +0000. [09:20:52] RECOVERY - itora.bunnies1230.com - LetsEncrypt on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Tue 20 Sep 2022 01:50:31 GMT +0000. [09:21:09] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.24, 7.75, 9.86 [09:21:27] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:21:29] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:21:54] RECOVERY - grk.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Sat 17 Sep 2022 21:13:49 GMT +0000. [09:22:09] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:22:16] RECOVERY - nintendo.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:22:22] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:45] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:22:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [09:23:00] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:01] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [09:23:14] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.316 second response time [09:23:16] [02mw-config] 07Universal-Omega closed pull request 03#4847: T9629: configure templateditor and extendedconfirmed rights for theredpionnerwiki - 13https://github.com/miraheze/mw-config/pull/4847 [09:23:17] [url] Page not found · GitHub · GitHub | github.com [09:23:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/63a9a3c245ca...11cbd0d07a9c [09:23:19] [url] Comparing 63a9a3c245ca...11cbd0d07a9c · miraheze/mw-config · GitHub | github.com [09:23:19] [02miraheze/mw-config] 07ugochimobi 0311cbd0d - T9629: configure templateditor and extendedconfirmed rights for theredpionnerwiki (#4847) [09:23:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.096 second response time [09:24:17] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.789 second response time [09:24:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.00, 2.46, 4.00 [09:24:22] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [09:24:29] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.105 second response time [09:24:33] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Wed 28 Sep 2022 05:13:53 GMT +0000. [09:24:37] miraheze/mw-config - Universal-Omega the build passed. [09:24:38] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:41] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.056 second response time [09:24:48] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [09:24:50] PROBLEM - az.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:52] RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:25:27] PROBLEM - wikipariksha.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:36] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:25:41] RECOVERY - wiki.puucraft.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.puucraft.net' will expire on Wed 26 Oct 2022 16:57:15 GMT +0000. [09:25:44] RECOVERY - www.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikiretet.ga' will expire on Tue 11 Oct 2022 04:42:43 GMT +0000. [09:25:57] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:26:21] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 14% [09:26:30] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:36] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [09:26:48] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 4% [09:27:01] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [09:27:07] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.055 second response time [09:27:26] !log [@test131] starting deploy of {'config': True} to all [09:27:27] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [09:27:37] PROBLEM - allthetropes.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:04] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Wed 24 Aug 2022 04:30:45 GMT +0000. [09:28:13] PROBLEM - 321nails.crpteam.club - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:15] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:18] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:28:24] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:28:29] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:48] RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:29:21] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [09:29:58] PROBLEM - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:45] PROBLEM - wiki.veloren.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:02] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.81, 2.41, 3.37 [09:32:36] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/14e4ac8a14f8 [09:32:50] [02miraheze/mw-config] 07Universal-Omega 0314e4ac8 - MirahezeFunctions: add function doc comments [09:32:51] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [09:32:52] [url] Page not found · GitHub · GitHub | github.com [09:32:53] [02mw-config] 07Universal-Omega opened pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [09:32:53] [url] Page not found · GitHub · GitHub | github.com [09:33:16] PROBLEM - creativecommons.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:25] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:47] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:33:47] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:04] miraheze/mw-config - Universal-Omega the build passed. [09:34:20] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.382 second response time [09:34:38] PROBLEM - iol.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:45] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:54] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:54] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/14e4ac8a14f8...abe74c1a2a17 [09:34:56] [url] Comparing 14e4ac8a14f8...abe74c1a2a17 · miraheze/mw-config · GitHub | github.com [09:34:57] [02miraheze/mw-config] 07Universal-Omega 03abe74c1 - Update .phpcs.xml [09:34:59] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [09:34:59] [url] Page not found · GitHub · GitHub | github.com [09:35:01] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:03] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:07] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:14] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:16] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:20] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:22] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:57] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [09:36:10] miraheze/mw-config - Universal-Omega the build has errored. [09:36:12] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [09:36:19] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:44] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [09:36:53] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/abe74c1a2a17...0fafcdf0776a [09:37:10] [url] Comparing abe74c1a2a17...0fafcdf0776a · miraheze/mw-config · GitHub | github.com [09:37:11] [02miraheze/mw-config] 07Universal-Omega 030fafcdf - Update .phpcs.xml [09:37:12] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [09:37:13] ... [09:37:16] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:26] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.011 second response time [09:37:28] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:53] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.729 second response time [09:37:54] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:10] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [09:38:16] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [09:38:18] miraheze/mw-config - Universal-Omega the build passed. [09:38:18] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.032 second response time [09:38:27] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:41] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.057 second response time [09:38:48] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.063 second response time [09:38:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 3 backends are down. mw121 mw122 mw132 [09:38:48] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [09:38:49] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:38:59] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [09:39:01] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [09:39:11] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.289 second response time [09:39:13] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.239 second response time [09:39:15] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.338 second response time [09:39:24] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [09:39:25] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.340 second response time [09:39:32] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.337 second response time [09:39:58] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:40:04] !log [@mwtask141] starting deploy of {'config': True} to all [09:40:05] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw121.miraheze.org [09:40:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [09:40:06] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:33] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:40:43] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [09:40:48] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 7% [09:40:58] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [09:40:59] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.066 second response time [09:41:04] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [09:41:59] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.046 second response time [09:42:01] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.244 second response time [09:42:31] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:36] RECOVERY - wikileague.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wikileague.cf' will expire on Mon 22 Aug 2022 22:21:46 GMT +0000. [09:42:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [09:43:01] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [09:43:18] PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:20] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:32] PROBLEM - wiki.minebox.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:44:04] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.317 second response time [09:44:09] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [09:44:18] PROBLEM - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:44:20] PROBLEM - fortis.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:23] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:45:25] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:52] PROBLEM - deepdark.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:16] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:46:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 3.62, 3.10 [09:47:09] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:16] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:47:23] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.045 second response time [09:48:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [09:48:09] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:16] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [09:48:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:03] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:05] PROBLEM - www.turtletown.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:08] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [09:49:47] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:51] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.72, 3.31 [09:50:34] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.18, 12.90, 8.91 [09:51:00] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Wed 24 Aug 2022 03:23:05 GMT +0000. [09:51:15] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.222 second response time [09:51:30] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [09:51:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:51:52] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:51:56] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.246 second response time [09:52:03] PROBLEM - source.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [09:52:09] PROBLEM - jp.popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:22] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:52:32] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Mon 22 Aug 2022 15:10:26 GMT +0000. [09:52:35] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 10.99, 8.66 [09:52:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:53:34] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.097 second response time [09:53:51] PROBLEM - taswin.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:54] RECOVERY - az.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'az.wikiretet.ga' will expire on Tue 11 Oct 2022 04:40:37 GMT +0000. [09:54:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.044 second response time [09:54:06] PROBLEM - uncyclopediakr.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:17] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:54:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.78, 3.39 [09:54:18] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Wed 24 Aug 2022 05:18:49 GMT +0000. [09:54:23] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.17, 8.31, 7.94 [09:54:39] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [09:55:11] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.079 second response time [09:55:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.08, 10.27, 8.03 [09:55:41] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [09:56:44] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:56:52] RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:56:56] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [09:57:17] RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Sat 01 Oct 2022 16:33:15 GMT +0000. [09:57:24] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [09:57:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 9.01, 7.84 [09:57:29] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:58:00] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:58:06] RECOVERY - 321nails.crpteam.club - LetsEncrypt on sslhost is OK: OK - Certificate '321nails.crpteam.club' will expire on Tue 25 Oct 2022 15:35:07 GMT +0000. [09:58:23] PROBLEM - agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:42] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.79, 1.54, 1.18 [09:58:59] RECOVERY - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Sun 30 Oct 2022 20:08:40 GMT +0000. [09:59:13] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [09:59:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.48, 2.59 [09:59:44] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 2 % full [10:00:04] RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Sat 15 Oct 2022 15:52:05 GMT +0000. [10:00:05] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [10:01:32] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:02:42] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.65, 1.59, 1.28 [10:03:03] RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Thu 27 Oct 2022 05:20:13 GMT +0000. [10:03:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 2.97, 2.62 [10:03:34] RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Thu 27 Oct 2022 14:58:25 GMT +0000. [10:03:55] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [10:04:15] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Tue 23 Aug 2022 01:47:24 GMT +0000. [10:04:22] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Tue 23 Aug 2022 08:10:17 GMT +0000. [10:04:31] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:29] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Tue 23 Aug 2022 09:28:23 GMT +0000. [10:06:46] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:49] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:06:56] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Tue 23 Aug 2022 01:40:20 GMT +0000. [10:06:58] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [10:07:04] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:07:19] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:07:25] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:30] PROBLEM - cp31 Stunnel HTTP for puppet141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [10:07:33] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [10:07:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:55] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:57] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [10:08:00] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:01] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:08:02] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:04] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:08:04] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:08:09] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:10] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [10:08:11] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:12] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:13] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:16] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:17] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.28, 2.66, 3.63 [10:08:19] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [10:08:19] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:25] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:31] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:38] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [10:08:48] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 54% [10:08:50] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.016 second response time [10:08:56] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:02] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [10:09:21] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:21] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.51, 5.12, 3.69 [10:09:23] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.071 second response time [10:09:24] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [10:09:26] RECOVERY - cp31 Stunnel HTTP for puppet141 on cp31 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.253 second response time [10:09:30] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:09:32] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.045 second response time [10:09:38] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:54] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [10:09:55] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:09:57] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.348 second response time [10:09:58] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.043 second response time [10:10:00] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [10:10:03] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [10:10:04] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [10:10:08] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:10:08] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.289 second response time [10:10:08] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.345 second response time [10:10:10] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [10:10:11] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:10:11] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [10:10:12] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [10:10:14] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [10:10:15] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [10:10:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.82, 2.11, 3.31 [10:10:18] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.222 second response time [10:10:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.247 second response time [10:10:28] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.325 second response time [10:10:29] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [10:10:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [10:10:50] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:11:07] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.032 second response time [10:11:21] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.26, 3.66, 3.33 [10:11:22] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [10:11:30] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.404 second response time [10:11:45] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [10:11:53] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.050 second response time [10:11:55] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 2% [10:12:07] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [10:12:08] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.064 second response time [10:12:29] PROBLEM - wiki.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [10:12:48] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 3% [10:12:53] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.014 second response time [10:13:02] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [10:13:22] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.074 second response time [10:13:24] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 6% [10:13:26] RECOVERY - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.phgalaxy.eu.org' will expire on Mon 10 Oct 2022 06:42:26 GMT +0000. [10:13:27] RECOVERY - wiki.minebox.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minebox.fr' will expire on Fri 16 Sep 2022 18:05:33 GMT +0000. [10:14:45] RECOVERY - deepdark.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deepdark.wiki' will expire on Mon 03 Oct 2022 20:17:33 GMT +0000. [10:14:45] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:02] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 3.220 second response time [10:15:04] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:13] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:28] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:16:01] PROBLEM - staratlaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:12] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 3.93, 3.69 [10:16:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.49, 8.25, 6.66 [10:17:37] PROBLEM - wiki.scpsharp.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:37] PROBLEM - wiki.projectdiablo2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:44] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.063 second response time [10:18:32] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [10:18:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.62, 7.78, 6.71 [10:19:40] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:13] PROBLEM - proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:25] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:41] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.083 second response time [10:20:56] PROBLEM - dkpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:00] RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Sat 17 Sep 2022 20:00:57 GMT +0000. [10:21:05] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [10:21:27] PROBLEM - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:41] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:21:45] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Tue 23 Aug 2022 02:37:13 GMT +0000. [10:21:51] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:22:13] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:46] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:23:17] PROBLEM - perpustaxaan.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:34] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.227 second response time [10:24:20] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:02] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:00] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:26:31] PROBLEM - www.famedata.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:33] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:12] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:30] RECOVERY - agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [10:27:31] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.037 second response time [10:27:31] PROBLEM - wiki.simorgh.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:57] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:28:01] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39169 bytes in 0.043 second response time [10:28:16] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:17] PROBLEM - kunwok.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:35] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Tue 23 Aug 2022 00:33:03 GMT +0000. [10:29:42] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.75, 2.80 [10:29:54] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:07] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:30:16] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:30:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 25.55, 15.08, 9.42 [10:30:42] PROBLEM - christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:19] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:21] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:33] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.13, 3.32, 2.75 [10:32:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [10:32:24] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:53] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:38] PROBLEM - rww2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [10:34:18] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:19] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:34:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.56, 9.69, 8.64 [10:34:38] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.053 second response time [10:35:19] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 24.47, 15.85, 10.33 [10:35:34] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:45] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:03] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:36:10] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.067 second response time [10:36:49] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.036 second response time [10:36:51] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:09] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:37:52] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.285 second response time [10:37:52] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 5 % full [10:37:57] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:07] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:32] PROBLEM - slymods.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:41] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.306 second response time [10:39:48] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:06] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:02] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.051 second response time [10:41:44] RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Mon 03 Oct 2022 22:23:06 GMT +0000. [10:41:47] RECOVERY - wiki.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erag.eu.org' will expire on Fri 16 Sep 2022 23:33:23 GMT +0000. [10:42:05] PROBLEM - smafallfest.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:09] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.150 second response time [10:42:09] PROBLEM - beta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:14] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.50, 3.91 [10:42:26] PROBLEM - test1.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:34] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:42:45] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 5.154 second response time [10:42:52] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:10] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.12, 9.89, 10.23 [10:44:01] RECOVERY - fortis.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:44:15] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [10:44:21] PROBLEM - unmade.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:44:38] RECOVERY - staratlaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'staratlaswiki.org' will expire on Mon 19 Sep 2022 03:34:09 GMT +0000. [10:44:39] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 12 Oct 2022 13:21:03 GMT +0000. [10:44:57] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:00] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Sat 01 Oct 2022 04:20:30 GMT +0000. [10:45:17] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:18] PROBLEM - ngsswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.11, 7.68, 9.38 [10:45:42] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:05] PROBLEM - pwsc.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:05] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 3.047 second response time [10:46:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.97, 4.00, 4.03 [10:46:18] RECOVERY - wiki.projectdiablo2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.projectdiablo2.com' will expire on Tue 11 Oct 2022 03:46:46 GMT +0000. [10:46:18] RECOVERY - wiki.scpsharp.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scpsharp.ga' will expire on Wed 31 Aug 2022 20:42:39 GMT +0000. [10:46:19] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [10:47:03] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.91, 3.99 [10:48:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:49:08] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Fri 02 Sep 2022 12:34:13 GMT +0000. [10:49:11] RECOVERY - proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [10:49:20] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:49:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [10:49:55] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.71, 4.13, 4.08 [10:50:18] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Mon 22 Aug 2022 22:30:31 GMT +0000. [10:50:37] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:51] RECOVERY - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssangyongsports.eu.org' will expire on Wed 31 Aug 2022 22:12:11 GMT +0000. [10:51:21] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Mon 10 Oct 2022 06:13:03 GMT +0000. [10:51:27] RECOVERY - ndg.nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Tue 23 Aug 2022 01:35:44 GMT +0000. [10:51:47] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Sat 24 Sep 2022 13:22:32 GMT +0000. [10:52:28] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [10:52:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.44, 11.00, 8.85 [10:52:40] RECOVERY - perpustaxaan.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'perpustaxaan.wiki' will expire on Tue 11 Oct 2022 04:34:44 GMT +0000. [10:53:21] RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [10:54:10] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Sat 03 Sep 2022 15:10:56 GMT +0000. [10:54:11] RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Fri 02 Sep 2022 12:38:19 GMT +0000. [10:54:25] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:26] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures [10:54:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 10.95, 9.10 [10:54:46] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:47] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.023 second response time [10:54:54] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:55:06] RECOVERY - www.famedata.org - LetsEncrypt on sslhost is OK: OK - Certificate 'famedata.org' will expire on Thu 29 Sep 2022 07:42:43 GMT +0000. [10:55:49] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:56:01] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:12] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Thu 15 Sep 2022 05:19:23 GMT +0000. [10:56:20] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:23] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.030 second response time [10:56:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.31, 11.60, 9.55 [10:56:41] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Sun 02 Oct 2022 10:55:38 GMT +0000. [10:56:41] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:56] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.020 second response time [10:57:31] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Wed 24 Aug 2022 04:55:08 GMT +0000. [10:57:36] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:57:50] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.025 second response time [10:58:15] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.017 second response time [10:58:35] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Mon 22 Aug 2022 23:35:07 GMT +0000. [10:58:40] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [10:59:04] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:59:10] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 24 Aug 2022 04:11:39 GMT +0000. [10:59:12] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Thu 01 Sep 2022 00:43:35 GMT +0000. [10:59:19] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Tue 23 Aug 2022 07:32:52 GMT +0000. [10:59:36] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [10:59:50] RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 17 Sep 2022 18:34:21 GMT +0000. [11:00:12] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.56, 3.02 [11:01:18] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.074 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [11:01:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.62, 9.46, 8.26 [11:01:31] RECOVERY - arcanewaterswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'arcanewaterswiki.org' will expire on Sun 30 Oct 2022 20:54:06 GMT +0000. [11:02:13] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 3.076 second response time [11:02:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.25, 3.39, 3.01 [11:03:13] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [11:03:15] RECOVERY - rww2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rww2.org' will expire on Sat 17 Sep 2022 22:30:25 GMT +0000. [11:03:32] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:34] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 23 Aug 2022 00:48:11 GMT +0000. [11:04:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.64, 10.64, 10.34 [11:05:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.287 second response time [11:05:43] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 5.127 second response time [11:05:45] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Tue 23 Aug 2022 08:56:40 GMT +0000. [11:05:50] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Tue 23 Aug 2022 01:29:10 GMT +0000. [11:06:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.61, 9.20, 9.87 [11:06:38] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:08:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.79, 1.26 [11:08:48] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [11:08:54] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.038 second response time [11:09:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.36, 10.46, 10.39 [11:09:58] PROBLEM - staff.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:19] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:10:24] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:34] PROBLEM - wiki.rtapp.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.44, 1.31, 1.15 [11:11:04] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:11:15] RECOVERY - smafallfest.org - LetsEncrypt on sslhost is OK: OK - Certificate 'smafallfest.org' will expire on Wed 26 Oct 2022 18:44:23 GMT +0000. [11:11:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.34, 8.45, 9.66 [11:11:25] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sat 01 Oct 2022 04:26:41 GMT +0000. [11:11:33] PROBLEM - wildbowiki.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:55] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Tue 23 Aug 2022 01:05:30 GMT +0000. [11:12:08] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:14] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [11:12:25] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:01] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [11:13:04] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:19] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.88, 9.48, 6.32 [11:13:36] PROBLEM - wiki.recaptime.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:40] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:49] PROBLEM - biblestrength.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:18] PROBLEM - wiki.helsinginpoytaroolipelaajat.fi - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:34] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:15:37] RECOVERY - pwsc.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwsc.polandballwiki.com' will expire on Sun 02 Oct 2022 10:41:20 GMT +0000. [11:15:37] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:16:12] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:42] RECOVERY - www.turtletown.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'www.turtletown.ca' will expire on Tue 20 Sep 2022 20:30:39 GMT +0000. [11:16:57] PROBLEM - miraheze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:17] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.31, 11.44, 7.89 [11:17:45] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.37, 3.96 [11:18:09] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Mon 22 Aug 2022 23:09:56 GMT +0000. [11:18:13] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:18:31] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [11:19:11] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:11] RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Tue 04 Oct 2022 18:55:51 GMT +0000. [11:19:16] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 6.70, 9.50, 7.60 [11:19:19] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:19:20] PROBLEM - polandball.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:25] PROBLEM - intp.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:14] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.331 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [11:20:25] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:32] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.035 second response time [11:21:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.67, 4.44, 4.15 [11:21:52] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:05] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:22:08] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:36] PROBLEM - www.ethene.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:38] PROBLEM - cloudwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:39] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:44] PROBLEM - zh.internetpedia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:47] PROBLEM - wiki.meower.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:50] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:53] PROBLEM - rarewiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:21] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:32] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:23:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:50] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.043 second response time [11:23:50] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:24] PROBLEM - ecole.science - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:25] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:24:54] PROBLEM - cp21 conntrack_table_size on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:25:11] PROBLEM - wiki.potabi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:20] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.054 second response time [11:25:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.86, 13.60, 10.64 [11:25:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.020 second response time [11:25:39] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.116 second response time [11:25:50] RECOVERY - arrieretoile.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'arrieretoile.fr' will expire on Thu 13 Oct 2022 04:37:30 GMT +0000. [11:25:52] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Mon 22 Aug 2022 15:10:26 GMT +0000. [11:26:01] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:10] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.032 second response time [11:26:13] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [11:26:36] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:37] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:39] PROBLEM - gp.ct777.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:46] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:51] PROBLEM - www.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:53] RECOVERY - cp21 conntrack_table_size on cp21 is OK: OK: nf_conntrack is 6 % full [11:27:04] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.278 second response time [11:27:06] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:06] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:11] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:17] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.04, 11.41, 10.18 [11:27:43] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:57] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:39] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:21] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.365 second response time [11:29:28] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:36] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:11] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:19] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:24] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:26] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:34] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:30:53] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.047 second response time [11:30:55] PROBLEM - wiki.fossbots.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:23] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.44, 11.03, 10.26 [11:31:41] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.025 second response time [11:32:19] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.215 second response time [11:32:25] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:26] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:09] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.037 second response time [11:34:52] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:05] PROBLEM - kiseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:26] PROBLEM - wiki.hsins.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:35] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [11:36:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.430 second response time [11:36:53] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:37:18] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:07] PROBLEM - reviews.thejoshmeister.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 25.65, 14.83, 9.95 [11:38:53] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.013 second response time [11:39:16] RECOVERY - staff.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'staff.polandballwiki.com' will expire on Sun 02 Oct 2022 10:43:17 GMT +0000. [11:39:25] RECOVERY - slymods.info - LetsEncrypt on sslhost is OK: OK - Certificate 'slymods.info' will expire on Fri 21 Oct 2022 06:31:32 GMT +0000. [11:39:57] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Tue 25 Oct 2022 07:00:25 GMT +0000. [11:40:06] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:41] RECOVERY - beta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:40:49] RECOVERY - test1.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:41:14] RECOVERY - wildbowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wildbowiki.net' will expire on Tue 06 Sep 2022 20:59:02 GMT +0000. [11:41:20] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:41:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.29, 10.37, 11.49 [11:41:35] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Mon 05 Sep 2022 00:02:52 GMT +0000. [11:41:57] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [11:42:13] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Tue 23 Aug 2022 08:05:34 GMT +0000. [11:42:16] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:40] RECOVERY - biblestrength.net - LetsEncrypt on sslhost is OK: OK - Certificate 'biblestrength.net' will expire on Wed 07 Sep 2022 18:16:09 GMT +0000. [11:42:45] RECOVERY - wiki.recaptime.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.tk' will expire on Wed 26 Oct 2022 23:10:27 GMT +0000. [11:42:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:26] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Mon 22 Aug 2022 23:01:19 GMT +0000. [11:43:36] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 23 Aug 2022 07:53:50 GMT +0000. [11:44:01] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:03] RECOVERY - wiki.helsinginpoytaroolipelaajat.fi - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.helsinginpoytaroolipelaajat.fi' will expire on Mon 29 Aug 2022 12:44:24 GMT +0000. [11:44:06] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:44:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.260 second response time [11:44:58] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 17 Sep 2022 21:39:52 GMT +0000. [11:45:10] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.016 second response time [11:45:11] RECOVERY - ngsswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ngsswiki.com' will expire on Sun 02 Oct 2022 05:32:39 GMT +0000. [11:45:14] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:45:33] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [11:45:35] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Tue 23 Aug 2022 01:13:17 GMT +0000. [11:46:05] PROBLEM - fortis.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:19] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:51] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 12 Oct 2022 13:21:03 GMT +0000. [11:47:05] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [11:47:14] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:20] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:47:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.32, 10.39, 10.93 [11:47:36] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:02] RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Sat 01 Oct 2022 16:22:56 GMT +0000. [11:48:02] PROBLEM - swests.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.14, 3.06, 4.00 [11:48:21] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [11:48:22] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:48:26] RECOVERY - polandball.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:48:37] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [11:48:38] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:45] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [11:48:59] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Wed 24 Aug 2022 03:23:05 GMT +0000. [11:49:12] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [11:49:21] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.043 second response time [11:49:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.16, 9.22, 10.36 [11:49:48] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Sun 02 Oct 2022 11:07:49 GMT +0000. [11:50:35] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.324 second response time [11:50:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.336 second response time [11:50:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.90, 11.44, 11.58 [11:50:51] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Tue 01 Nov 2022 02:48:17 GMT +0000. [11:50:55] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.083 second response time [11:50:56] PROBLEM - dkpedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:05] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw131 [11:51:21] RECOVERY - wiki.meower.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meower.org' will expire on Thu 27 Oct 2022 03:39:58 GMT +0000. [11:51:23] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.321 second response time [11:51:41] RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Sat 24 Sep 2022 13:58:45 GMT +0000. [11:51:45] RECOVERY - cloudwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cloudwiki.org' will expire on Sun 23 Oct 2022 16:49:57 GMT +0000. [11:51:54] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:52:02] RECOVERY - zh.internetpedia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.internetpedia.tk' will expire on Mon 22 Aug 2022 22:07:13 GMT +0000. [11:52:09] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:11] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:52:21] RECOVERY - rarewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewiki.org' will expire on Wed 28 Sep 2022 05:06:19 GMT +0000. [11:52:41] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:53:24] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.99, 7.98, 9.71 [11:53:30] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [11:54:02] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:04] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:04] RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Mon 22 Aug 2022 21:07:09 GMT +0000. [11:54:09] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:09] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:09] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:13] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [11:54:13] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:26] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [11:54:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 3.62, 7.02, 9.70 [11:54:37] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Mon 10 Oct 2022 06:24:27 GMT +0000. [11:54:47] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.868 second response time [11:54:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw121 mw122 mw132 mw141 mw142 [11:54:52] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:56] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [11:55:00] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:55:12] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.046 second response time [11:55:17] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Sat 03 Sep 2022 15:10:56 GMT +0000. [11:55:18] PROBLEM - wikipariksha.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:23] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [11:55:24] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Sat 01 Oct 2022 16:25:42 GMT +0000. [11:55:31] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 44% [11:55:44] RECOVERY - www.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikiretet.ga' will expire on Tue 11 Oct 2022 04:42:43 GMT +0000. [11:56:01] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:56:02] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Sat 17 Sep 2022 22:20:19 GMT +0000. [11:56:02] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [11:56:03] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [11:56:04] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:56:05] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:08] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [11:56:09] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.323 second response time [11:56:12] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [11:56:14] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:56:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.73, 1.77, 3.10 [11:56:25] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [11:56:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.320 second response time [11:56:44] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Tue 23 Aug 2022 03:02:35 GMT +0000. [11:56:59] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Wed 24 Aug 2022 02:28:53 GMT +0000. [11:57:04] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:04] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:57:05] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 07 Oct 2022 11:49:47 GMT +0000. [11:57:26] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 32% [11:57:53] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:02] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Mon 26 Sep 2022 13:36:29 GMT +0000. [11:59:03] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:03] PROBLEM - cp30 Stunnel HTTP for test131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:05] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:09] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:09] PROBLEM - cp30 Stunnel HTTP for puppet141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:20] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.031 second response time [11:59:27] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:33] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [11:59:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:07] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 5.071 second response time [12:00:12] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.323 second response time [12:00:17] RECOVERY - gametheorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'gametheorywiki.org' will expire on Sun 25 Sep 2022 20:40:02 GMT +0000. [12:00:19] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 3.034 second response time [12:00:20] RECOVERY - wiki.fossbots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fossbots.org' will expire on Sun 16 Oct 2022 13:11:42 GMT +0000. [12:00:21] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Wed 24 Aug 2022 04:17:42 GMT +0000. [12:00:41] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:59] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [12:01:01] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [12:01:04] RECOVERY - cp30 Stunnel HTTP for test131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.328 second response time [12:01:05] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99038 bytes in 0.819 second response time [12:01:10] RECOVERY - cp30 Stunnel HTTP for puppet141 on cp30 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.256 second response time [12:01:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.272 second response time [12:01:14] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.058 second response time [12:01:27] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [12:01:37] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.035 second response time [12:01:42] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.074 second response time [12:01:42] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.015 second response time [12:02:02] RECOVERY - ipv6bolivia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'ipv6bolivia.tk' will expire on Tue 23 Aug 2022 13:09:22 GMT +0000. [12:02:02] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.314 second response time [12:02:07] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [12:02:13] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.032 second response time [12:02:54] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sun 02 Oct 2022 10:22:40 GMT +0000. [12:02:54] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.336 second response time [12:03:14] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [12:03:30] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:35] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [12:03:58] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:03] PROBLEM - n64brew.dev - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:09] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:19] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [12:04:50] RECOVERY - wiki.hsins.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hsins.eu' will expire on Wed 05 Oct 2022 21:47:47 GMT +0000. [12:05:03] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:03] PROBLEM - famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:05] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [12:05:54] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:00] PROBLEM - wiki.alathramc.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:49] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Tue 23 Aug 2022 02:51:23 GMT +0000. [12:07:05] PROBLEM - wiki.worldbuilding.network - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:06] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:31] PROBLEM - tensegritywiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:37] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [12:07:48] RECOVERY - reviews.thejoshmeister.com - LetsEncrypt on sslhost is OK: OK - Certificate 'reviews.thejoshmeister.com' will expire on Thu 29 Sep 2022 08:42:34 GMT +0000. [12:08:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:08:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.47, 3.01 [12:08:48] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [12:08:56] RECOVERY - wiki.rtapp.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rtapp.tk' will expire on Wed 26 Oct 2022 23:42:41 GMT +0000. [12:09:07] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:10:54] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:11:31] PROBLEM - wiki.evscope.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:44] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 14 day(s) (Fri 19 Aug 2022 17:20:05 GMT +0000). [12:11:44] PROBLEM - thenintendowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:19] PROBLEM - beta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.36, 10.65, 8.44 [12:12:54] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:12:55] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.0002506971359 secs [12:13:43] PROBLEM - phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:49] PROBLEM - wiki.iurevar.world - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.63, 9.50, 8.35 [12:13:50] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:35] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:57] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.034 second response time [12:15:15] PROBLEM - www.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:18] PROBLEM - en.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.92, 12.92, 9.72 [12:15:47] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:15:50] RECOVERY - fortis.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:16:01] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:03] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:03] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:21] PROBLEM - squareenix.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:34] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:16:38] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.026 second response time [12:16:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.45, 9.36, 8.61 [12:17:01] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 11.14, 9.47 [12:17:47] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:17:50] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:54] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [12:17:59] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:18:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.93, 3.62 [12:18:32] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39177 bytes in 0.034 second response time [12:18:36] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:45] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 1.047 second response time [12:18:50] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:03] PROBLEM - donkeykong.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:04] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:05] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.029 second response time [12:19:26] PROBLEM - buildabearwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.59, 9.66, 9.12 [12:19:33] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:49] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:19:59] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 5.055 second response time [12:20:07] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:21] PROBLEM - smashloop.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:34] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:40] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:04] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:21:30] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:22:11] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:22:12] PROBLEM - wiki.exputra.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 4.09, 3.74 [12:22:22] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:22:44] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 2.058 second response time [12:23:24] PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:29] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:24:32] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.057 second response time [12:25:32] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [12:25:40] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [12:25:48] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [12:25:51] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:17] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:39] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:04] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Wed 19 Oct 2022 06:37:38 GMT +0000. [12:27:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.01, 10.58, 9.63 [12:27:50] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:01] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:09] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [12:28:33] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:28:37] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:40] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:03] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:28] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:39] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.43, 3.47, 3.03 [12:29:56] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:30:04] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [12:30:37] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 9706 MB (25% inode=96%); [12:31:00] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:31:10] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.020 second response time [12:31:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.52, 12.18, 10.51 [12:31:43] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.11, 2.61, 2.77 [12:31:51] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.061 second response time [12:32:39] RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Tue 23 Aug 2022 09:10:11 GMT +0000. [12:32:42] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.032 second response time [12:32:56] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Tue 23 Aug 2022 01:21:14 GMT +0000. [12:33:04] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.027 second response time [12:33:07] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:33:13] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.050 second response time [12:33:14] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Tue 23 Aug 2022 01:47:24 GMT +0000. [12:33:39] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Wed 24 Aug 2022 01:37:17 GMT +0000. [12:34:12] PROBLEM - wiki.slimeapedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.64, 3.81 [12:34:23] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:34:39] PROBLEM - wiki.zergmafia.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:43] RECOVERY - kiseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kiseki.wiki' will expire on Wed 24 Aug 2022 06:15:40 GMT +0000. [12:34:52] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:34:52] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Mon 22 Aug 2022 21:48:31 GMT +0000. [12:34:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.30, 0.43, 0.54 [12:35:01] RECOVERY - wiki.alathramc.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.alathramc.com' will expire on Tue 20 Sep 2022 12:13:47 GMT +0000. [12:35:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.98, 9.34, 6.17 [12:35:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.16, 10.59, 10.39 [12:35:35] PROBLEM - wiki.lu9.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:43] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.087 second response time [12:35:45] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:25] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.070 second response time [12:36:56] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:37:00] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:07] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.28, 8.91, 6.44 [12:37:09] PROBLEM - wikipanel.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:09] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:17] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 8.829 second response time [12:37:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.74, 8.41, 9.59 [12:37:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.44, 2.44, 1.66 [12:37:57] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 24.31, 15.36, 10.66 [12:39:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 6.470 second response time [12:39:11] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [12:39:18] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.667 second response time [12:39:24] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:39:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.42, 3.00 [12:39:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.54, 1.79, 1.51 [12:40:00] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.630 second response time [12:40:07] RECOVERY - wiki.evscope.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Fri 07 Oct 2022 11:21:52 GMT +0000. [12:40:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.64, 3.69 [12:40:40] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:46:28 GMT +0000. [12:40:42] RECOVERY - thenintendowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thenintendowiki.org' will expire on Sat 22 Oct 2022 05:06:23 GMT +0000. [12:41:00] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:41:17] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.303 second response time [12:41:22] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.651 second response time [12:41:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.10, 2.57, 2.74 [12:41:30] RECOVERY - beta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:41:36] RECOVERY - feathercare.net - LetsEncrypt on sslhost is OK: OK - Certificate 'feathercare.net' will expire on Sat 27 Aug 2022 14:00:44 GMT +0000. [12:42:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.52, 3.64 [12:42:25] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:42:48] RECOVERY - wiki.iurevar.world - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.iurevar.world' will expire on Wed 21 Sep 2022 08:53:59 GMT +0000. [12:42:59] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [12:43:03] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:31] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [12:43:40] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [12:43:43] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:16] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Tue 23 Aug 2022 08:34:15 GMT +0000. [12:44:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.36, 0.85, 1.17 [12:44:26] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:44:29] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:33] RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Sat 17 Sep 2022 20:10:50 GMT +0000. [12:44:42] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [12:44:43] RECOVERY - en.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiretet.ga' will expire on Sun 02 Oct 2022 05:37:01 GMT +0000. [12:44:43] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [12:45:02] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.053 second response time [12:45:02] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [12:45:18] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:45] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:45:49] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:15] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.04, 2.50, 3.24 [12:46:26] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:26] RECOVERY - swests.org - LetsEncrypt on sslhost is OK: OK - Certificate 'swests.org' will expire on Thu 20 Oct 2022 03:33:41 GMT +0000. [12:46:29] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:29] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:38] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:46] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 24 Aug 2022 05:02:45 GMT +0000. [12:46:54] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.281 second response time [12:46:56] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:47:13] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Wed 24 Aug 2022 05:33:46 GMT +0000. [12:47:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 5.786 second response time [12:47:39] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.308 second response time [12:48:07] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [12:48:11] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [12:48:24] RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:48:25] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [12:48:27] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [12:48:29] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.337 second response time [12:48:38] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:48:39] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.337 second response time [12:48:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [12:49:08] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw132 mw141 [12:49:10] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Mon 22 Aug 2022 23:09:56 GMT +0000. [12:49:15] RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Tue 04 Oct 2022 18:55:51 GMT +0000. [12:49:24] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.059 second response time [12:49:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 10.18, 9.27 [12:49:56] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [12:50:08] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [12:50:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 3.92, 3.63 [12:50:45] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.042 second response time [12:50:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [12:51:08] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [12:51:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.57, 9.09, 8.98 [12:51:46] RECOVERY - wiki.exputra.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.exputra.com' will expire on Fri 16 Sep 2022 18:09:57 GMT +0000. [12:51:47] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [12:51:49] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 3.061 second response time [12:51:54] RECOVERY - grk.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Sat 17 Sep 2022 21:13:49 GMT +0000. [12:52:50] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [12:53:10] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Wed 24 Aug 2022 05:18:49 GMT +0000. [12:54:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.99, 1.43 [12:54:47] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:54:53] PROBLEM - www.proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:55:24] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [12:55:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 18.66, 13.10, 10.52 [12:55:50] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:14] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:56:36] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.49, 1.31 [12:56:48] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [12:56:59] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:57:05] PROBLEM - pokemon2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:09] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [12:57:13] PROBLEM - commonwealth.0um.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:42] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:57:46] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.045 second response time [12:58:02] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 24 Aug 2022 04:11:00 GMT +0000. [12:58:03] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:04] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:04] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [12:58:17] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:27] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:58:30] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.071 second response time [12:58:30] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.23, 8.72, 11.22 [12:58:44] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:47] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:03] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:59:49] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.068 second response time [13:00:02] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.418 second response time [13:00:04] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [13:00:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:00:30] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:01:20] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:24] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:29] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:40] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:34] PROBLEM - thepolicyhub.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.31, 6.02, 9.51 [13:02:47] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:10] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:11] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:03:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.87, 10.82, 11.24 [13:03:39] PROBLEM - civclassic.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:43] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.026 second response time [13:03:49] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:55] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [13:04:18] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:37] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 23 Aug 2022 00:48:11 GMT +0000. [13:04:38] RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zergmafia.top' will expire on Tue 06 Sep 2022 20:57:49 GMT +0000. [13:04:55] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:39] PROBLEM - wiki.auraxis.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:50] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Tue 23 Aug 2022 01:29:10 GMT +0000. [13:05:53] RECOVERY - wikipanel.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipanel.org' will expire on Sun 02 Oct 2022 04:51:38 GMT +0000. [13:05:54] PROBLEM - phytinfo.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:05] PROBLEM - wiki.hsins.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:10] PROBLEM - wiki.mostly.vet - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:14] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:16] PROBLEM - atnarsia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:22] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Wed 24 Aug 2022 03:18:35 GMT +0000. [13:06:30] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Tue 23 Aug 2022 08:10:17 GMT +0000. [13:06:30] PROBLEM - ping6 on cp21 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 2.68 ms [13:06:33] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:59] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [13:07:09] RECOVERY - wiki.worldbuilding.network - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Mon 10 Oct 2022 05:23:51 GMT +0000. [13:07:32] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.17, 2.88 [13:08:27] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:31] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.103 second response time [13:09:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.89, 8.49, 9.97 [13:09:47] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.037 second response time [13:09:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.05, 3.17, 2.92 [13:09:50] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.187 second response time [13:09:50] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 2.043 second response time [13:09:56] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:23] PROBLEM - www.sidem.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:41] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:10:59] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:36] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [13:13:09] PROBLEM - beta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:15] PROBLEM - biblestrength.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.28, 13.45, 11.66 [13:13:36] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:57] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:29] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [13:15:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.60, 9.58, 10.46 [13:15:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:43] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:15:44] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:45] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:50] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:15:50] PROBLEM - cp31 Stunnel HTTP for mwtask141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:15:51] PROBLEM - mail121 PowerDNS Recursor on mail121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:15:52] hah, i was about to say [13:15:53] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:57] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:15:57] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:15:58] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:01] something has gone down [13:16:04] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:16:06] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:08] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:12] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:16:17] PROBLEM - cp30 Stunnel HTTP for mwtask141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:16:18] PROBLEM - gluster122 PowerDNS Recursor on gluster122 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:16:19] PROBLEM - cp30 Stunnel HTTP for phab121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:16:26] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:27] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:16:35] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:16:37] PROBLEM - wiki.minebox.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:38] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:41] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:43] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:44] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:48] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [13:16:54] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:54] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:01] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:17:09] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [13:17:18] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.27, 8.48, 9.92 [13:17:26] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:28] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 2.32, 1.89, 0.91 [13:17:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.291 second response time [13:17:39] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:42] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [13:17:42] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [13:17:42] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [13:17:44] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.333 second response time [13:17:48] RECOVERY - cp31 Stunnel HTTP for mwtask141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.323 second response time [13:17:49] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.81, 3.54, 2.48 [13:17:50] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.311 second response time [13:17:50] RECOVERY - mail121 PowerDNS Recursor on mail121 is OK: DNS OK: 0.068 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [13:17:56] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.317 second response time [13:17:56] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [13:17:57] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.337 second response time [13:17:58] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 43% [13:18:01] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.397 second response time [13:18:04] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.336 second response time [13:18:06] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.236 second response time [13:18:07] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.308 second response time [13:18:14] PROBLEM - ngsswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:14] RECOVERY - gluster122 PowerDNS Recursor on gluster122 is OK: DNS OK: 0.215 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [13:18:16] RECOVERY - cp30 Stunnel HTTP for mwtask141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.338 second response time [13:18:18] RECOVERY - cp30 Stunnel HTTP for phab121 on cp30 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.313 second response time [13:18:25] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.019 second response time [13:18:26] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [13:18:35] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.353 second response time [13:18:41] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.321 second response time [13:18:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:43] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.183 second response time [13:18:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [13:18:55] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.029 second response time [13:19:00] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:10] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.019 second response time [13:19:14] PROBLEM - wikifencing.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:26] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.095 second response time [13:19:28] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.62, 1.37, 0.83 [13:19:34] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.017 second response time [13:19:41] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 88% [13:19:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 0.89, 1.64, 3.66 [13:19:49] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 0.57, 2.42, 2.20 [13:19:53] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:24] PROBLEM - wiki.wisrail.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:40] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [13:20:48] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 2% [13:20:55] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:20:57] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:21:06] RECOVERY - ping6 on cp21 is OK: PING OK - Packet loss = 0%, RTA = 2.64 ms [13:21:09] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.054 second response time [13:21:09] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [13:21:23] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.064 second response time [13:21:23] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:30] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:21:41] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 6% [13:21:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 2.59, 3.75 [13:22:08] PROBLEM - yokaiwatchwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:24] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [13:22:28] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:36] PROBLEM - id.altilunium.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:07] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 7% [13:23:18] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:51] PROBLEM - taswin.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:53] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:24:02] RECOVERY - www.proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [13:24:05] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [13:24:17] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [13:24:21] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 21.46, 12.39, 9.40 [13:24:40] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.313 second response time [13:25:17] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.042 second response time [13:25:18] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:46] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:26:06] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.039 second response time [13:26:14] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:30] RECOVERY - commonwealth.0um.io - LetsEncrypt on sslhost is OK: OK - Certificate 'commonwealth.0um.io' will expire on Thu 20 Oct 2022 18:21:09 GMT +0000. [13:27:13] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:26] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:28] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 8.549 second response time [13:27:49] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [13:27:50] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:27:51] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Wed 24 Aug 2022 04:55:08 GMT +0000. [13:27:55] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:27:59] RECOVERY - arrieretoile.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'arrieretoile.fr' will expire on Thu 13 Oct 2022 04:37:30 GMT +0000. [13:28:11] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [13:28:19] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.020 second response time [13:28:23] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Wed 28 Sep 2022 05:13:53 GMT +0000. [13:28:23] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:40] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 1.027 second response time [13:28:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.03, 1.74, 1.29 [13:29:09] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:14] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:28] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.344 second response time [13:29:36] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.210 second response time [13:29:39] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:56] PROBLEM - animatedfeet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:00] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.268 second response time [13:30:03] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.012 second response time [13:30:04] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.060 second response time [13:30:10] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 15 Sep 2022 05:33:41 GMT +0000. [13:30:22] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [13:30:25] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sat 17 Sep 2022 20:22:48 GMT +0000. [13:30:42] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.44, 1.63, 1.30 [13:31:11] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:31:39] RECOVERY - mabi.myomi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'mabi.myomi.net' will expire on Tue 23 Aug 2022 13:11:15 GMT +0000. [13:31:50] RECOVERY - wiki.slimeapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimeapedia.org' will expire on Mon 10 Oct 2022 05:19:54 GMT +0000. [13:32:11] PROBLEM - correypedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:26] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [13:32:26] RECOVERY - thepolicyhub.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thepolicyhub.org.uk' will expire on Tue 23 Aug 2022 02:56:36 GMT +0000. [13:32:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 11.94, 10.86 [13:33:13] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [13:33:19] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:33:27] RECOVERY - ipv6bolivia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'ipv6bolivia.tk' will expire on Tue 23 Aug 2022 13:09:22 GMT +0000. [13:34:25] RECOVERY - wiki.auraxis.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.auraxis.co' will expire on Fri 21 Oct 2022 06:34:16 GMT +0000. [13:34:25] RECOVERY - wiki.lu9.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lu9.io' will expire on Tue 06 Sep 2022 20:59:54 GMT +0000. [13:34:31] PROBLEM - roblox-wiki.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:53] RECOVERY - phytinfo.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'phytinfo.cf' will expire on Thu 27 Oct 2022 15:20:29 GMT +0000. [13:35:28] RECOVERY - wiki.hsins.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.hsins.eu' will expire on Wed 05 Oct 2022 21:47:47 GMT +0000. [13:35:45] RECOVERY - wiki.mostly.vet - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mostly.vet' will expire on Fri 02 Sep 2022 12:38:42 GMT +0000. [13:36:00] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Tue 23 Aug 2022 07:52:18 GMT +0000. [13:36:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.64, 8.89, 9.92 [13:36:48] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:54] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:59] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [13:37:47] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:48] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:55] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:08] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:38:12] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [13:38:17] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:22] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:38:28] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:31] Shutter down [13:38:42] PROBLEM - cp30 Stunnel HTTP for matomo131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:38:45] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw142 [13:38:52] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.158 second response time [13:39:10] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw121 mw131 mw141 mw142 [13:39:33] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:48] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:39:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.160 second response time [13:39:51] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Mon 22 Aug 2022 22:51:43 GMT +0000. [13:40:04] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:40:07] RECOVERY - www.sidem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.sidem.wiki' will expire on Tue 23 Aug 2022 12:59:12 GMT +0000. [13:40:19] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 14 day(s) (Fri 19 Aug 2022 17:20:05 GMT +0000). [13:40:25] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [13:40:38] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 56% [13:41:03] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.012 second response time [13:41:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.303 second response time [13:41:46] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 11% [13:41:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [13:41:53] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:06] RECOVERY - biblestrength.net - LetsEncrypt on sslhost is OK: OK - Certificate 'biblestrength.net' will expire on Wed 07 Sep 2022 18:16:09 GMT +0000. [13:42:20] RECOVERY - beta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:42:20] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [13:42:37] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 39% [13:42:59] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [13:43:02] RECOVERY - wikileague.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wikileague.cf' will expire on Mon 22 Aug 2022 22:21:46 GMT +0000. [13:43:04] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:05] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:06] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:52:26] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [13:52:27] PROBLEM - jp.popira.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:30] RECOVERY - gluster102 PowerDNS Recursor on gluster102 is OK: DNS OK: 2.300 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [13:52:33] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.045 second response time [13:52:36] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.319 second response time [13:52:38] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 3.064 second response time [13:52:39] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:42] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.346 second response time [13:52:47] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.435 second response time [13:52:47] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:52:48] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:52:49] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.195 second response time [13:52:54] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 1.036 second response time [13:53:00] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 2.091 second response time [13:53:21] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.221 second response time [13:53:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:28] PROBLEM - wiki.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:33] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 7.069 second response time [13:53:40] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [13:53:50] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.806 second response time [13:54:04] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [13:54:04] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:54:12] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 5 % full [13:54:21] PROBLEM - rct.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.92, 9.41, 7.49 [13:54:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [13:55:08] PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:10] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [13:55:26] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:55:28] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.146 second response time [13:55:32] PROBLEM - www.proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:48] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Mon 22 Aug 2022 23:00:39 GMT +0000. [13:56:02] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 3% [13:56:16] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [13:56:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.97, 8.01, 7.21 [13:56:51] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [13:56:59] PROBLEM - patternarchive.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:13] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:24] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:57:34] PROBLEM - wikien.wildterra2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:46] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.276 second response time [13:57:54] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:04] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:07] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.185 second response time [13:58:27] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [13:58:47] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:58:59] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:17] PROBLEM - wiki.veloren.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.76, 2.44 [13:59:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.33, 10.71, 7.42 [13:59:41] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:41] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:52] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:08] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:23] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.727 second response time [14:00:35] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:01] PROBLEM - entw.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:14] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.053 second response time [14:01:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.97, 2.30 [14:01:24] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.204 second response time [14:01:48] RECOVERY - correypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'correypedia.org' will expire on Thu 01 Sep 2022 00:41:01 GMT +0000. [14:01:49] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:05] PROBLEM - cp21 Stunnel HTTP for puppet141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:08] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:07] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.66, 9.25, 6.51 [14:03:15] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.41, 8.63, 7.40 [14:03:33] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:59] RECOVERY - roblox-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'roblox-wiki.tk' will expire on Thu 27 Oct 2022 15:10:45 GMT +0000. [14:04:08] RECOVERY - cp21 Stunnel HTTP for puppet141 on cp21 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.014 second response time [14:04:09] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:18] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:18] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:37] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:04:37] po [14:04:48] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.010 second response time [14:05:07] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 9.89, 9.71, 7.03 [14:05:25] PROBLEM - civclassic.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:33] RECOVERY - atnarsia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'atnarsia.com' will expire on Sun 25 Sep 2022 20:45:25 GMT +0000. [14:05:42] PROBLEM - txt-dex.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:04] PROBLEM - www.allthetropes.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:41] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 9326 MB (24% inode=96%); [14:07:09] PROBLEM - wikipanel.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:09] PROBLEM - history.estill.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:29] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:48] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [14:07:57] PROBLEM - scum-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:01] PROBLEM - wiki.milkywayidle.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:03] PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:07] PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:24] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:08:43] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:09:11] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.040 second response time [14:09:19] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:10] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.72, 3.15 [14:10:13] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:25] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.319 second response time [14:10:30] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:10:34] PROBLEM - discordian.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:17] PROBLEM - cnt.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:40] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.40, 2.60, 1.43 [14:11:49] PROBLEM - arru.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:16] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:18] PROBLEM - reviwiki.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:24] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.121 second response time [14:12:32] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:37] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:13:04] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.108 second response time [14:13:27] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [14:13:45] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:14:43] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:50] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [14:15:23] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [14:15:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 14.49, 11.56, 7.45 [14:15:57] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.233 second response time [14:16:25] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [14:16:27] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:16:30] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.71, 1.49, 1.26 [14:16:53] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [14:16:56] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.095 second response time [14:17:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [14:17:39] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:17:43] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sat 17 Sep 2022 22:00:09 GMT +0000. [14:17:48] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.71, 8.52, 6.84 [14:17:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:21] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 11.48, 8.49, 6.08 [14:18:23] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:18:27] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Sat 01 Oct 2022 16:09:21 GMT +0000. [14:18:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.82, 14.18, 9.40 [14:18:51] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Wed 24 Aug 2022 04:03:58 GMT +0000. [14:19:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 2.93, 2.42 [14:19:40] RECOVERY - fortis.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:19:47] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.58, 0.73, 0.60 [14:19:50] RECOVERY - yokaiwatchwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yokaiwatchwiki.com' will expire on Tue 01 Nov 2022 02:48:17 GMT +0000. [14:19:50] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3401 bytes in 7.284 second response time [14:20:24] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.074 second response time [14:20:24] RECOVERY - wiki.wisrail.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wisrail.de' will expire on Tue 06 Sep 2022 21:01:07 GMT +0000. [14:20:37] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [14:20:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.86, 11.53, 9.03 [14:20:42] RECOVERY - id.altilunium.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'id.altilunium.xyz' will expire on Fri 23 Sep 2022 00:33:37 GMT +0000. [14:20:49] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:20:54] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:14] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.90, 2.98, 2.51 [14:21:33] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:21:54] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:22:01] RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:22:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 8.38, 8.85, 6.84 [14:22:35] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.70, 9.63, 8.63 [14:22:49] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.039 second response time [14:23:28] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.042 second response time [14:23:37] RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Mon 22 Aug 2022 21:33:31 GMT +0000. [14:23:49] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Wed 24 Aug 2022 03:34:26 GMT +0000. [14:24:10] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Sat 03 Sep 2022 15:10:56 GMT +0000. [14:24:48] RECOVERY - pokemon2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:25:40] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.048 second response time [14:26:06] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:14] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Wed 24 Aug 2022 01:30:15 GMT +0000. [14:26:37] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:26:49] RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Sat 17 Sep 2022 18:49:33 GMT +0000. [14:27:02] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [14:28:07] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:28:27] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:28:36] RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Sat 15 Oct 2022 15:52:05 GMT +0000. [14:28:36] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 9101 MB (23% inode=96%); [14:28:53] RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:28:55] RECOVERY - arrieretoile.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'arrieretoile.fr' will expire on Thu 13 Oct 2022 04:37:30 GMT +0000. [14:28:59] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Wed 24 Aug 2022 04:17:42 GMT +0000. [14:29:11] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:29:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.91, 3.09 [14:29:38] [02mw-config] 07ugochimobi opened pull request 03#4849: T9629: make editextendedconfirmedprotected and edittemplateprotected available in theredpionnerwiki - 13https://github.com/miraheze/mw-config/pull/4849 [14:29:38] [url] Page not found · GitHub · GitHub | github.com [14:29:40] RECOVERY - wiki.danvs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Tue 23 Aug 2022 02:16:10 GMT +0000. [14:29:40] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [14:29:59] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Wed 28 Sep 2022 05:13:53 GMT +0000. [14:30:18] RECOVERY - entw.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'entw.ml' will expire on Tue 01 Nov 2022 04:38:34 GMT +0000. [14:30:25] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Sat 01 Oct 2022 16:16:26 GMT +0000. [14:30:31] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.086 second response time [14:30:33] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:30:35] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.287 second response time [14:30:43] miraheze/mw-config - ugochimobi the build passed. [14:30:50] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sat 17 Sep 2022 20:22:48 GMT +0000. [14:31:09] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:14] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:31:14] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 3.17, 2.91 [14:31:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.32, 13.13, 9.72 [14:31:33] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:00] RECOVERY - gametheorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'gametheorywiki.org' will expire on Sun 25 Sep 2022 20:40:02 GMT +0000. [14:32:08] PROBLEM - cp20 NTP time on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:34] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sat 17 Sep 2022 18:42:17 GMT +0000. [14:32:35] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.023 second response time [14:32:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.72, 9.73, 8.70 [14:32:54] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sun 02 Oct 2022 10:22:40 GMT +0000. [14:33:11] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.017 second response time [14:33:12] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.023 second response time [14:33:12] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Tue 23 Aug 2022 01:47:24 GMT +0000. [14:33:12] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [14:33:25] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Tue 23 Aug 2022 01:42:45 GMT +0000. [14:34:07] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset -0.0001504123211 secs [14:34:30] RECOVERY - txt-dex.com - LetsEncrypt on sslhost is OK: OK - Certificate 'txt-dex.com' will expire on Sat 29 Oct 2022 14:13:35 GMT +0000. [14:34:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.04, 8.65, 8.41 [14:35:05] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:35:09] RECOVERY - www.allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 23 Aug 2022 01:54:39 GMT +0000. [14:35:21] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [14:35:41] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 1.040 second response time [14:35:50] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Wed 24 Aug 2022 03:29:39 GMT +0000. [14:35:50] RECOVERY - wikipanel.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipanel.org' will expire on Sun 02 Oct 2022 04:51:38 GMT +0000. [14:37:05] RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Mon 29 Aug 2022 18:06:45 GMT +0000. [14:37:07] PROBLEM - cp20 ferm_active on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:37:13] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw142 [14:37:25] RECOVERY - wiki.milkywayidle.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.milkywayidle.com' will expire on Fri 21 Oct 2022 06:32:46 GMT +0000. [14:38:21] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Wed 26 Oct 2022 20:54:04 GMT +0000. [14:38:54] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [14:39:08] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Wed 24 Aug 2022 03:08:19 GMT +0000. [14:39:13] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [14:39:25] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:44] RECOVERY - discordian.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'discordian.wiki' will expire on Fri 16 Sep 2022 18:03:24 GMT +0000. [14:39:54] PROBLEM - wiki.virtualjet.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:04] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:40:17] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:52] RECOVERY - feathercare.net - LetsEncrypt on sslhost is OK: OK - Certificate 'feathercare.net' will expire on Sat 27 Aug 2022 14:00:44 GMT +0000. [14:41:18] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:41:21] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:27] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [14:41:56] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:17] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sat 17 Sep 2022 19:46:36 GMT +0000. [14:42:20] RECOVERY - cp20 ferm_active on cp20 is OK: OK ferm input default policy is set [14:42:43] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:20] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.044 second response time [14:43:26] PROBLEM - biblestrength.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:31] PROBLEM - slupe.fp64.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:44:25] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:36] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:40] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.065 second response time [14:44:57] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:04] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Wed 24 Aug 2022 04:41:20 GMT +0000. [14:45:56] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:03] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:03] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:36] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:56] RECOVERY - wikifencing.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wikifencing.ru' will expire on Tue 06 Sep 2022 20:55:14 GMT +0000. [14:47:04] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:47:28] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.059 second response time [14:47:59] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:48:20] PROBLEM - swests.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:31] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [14:48:31] PROBLEM - wiki.mcpirevival.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:44] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [14:48:57] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:18] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:19] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 6 % full [14:49:19] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:49:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.44, 10.39, 11.45 [14:49:37] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Wed 31 Aug 2022 20:55:03 GMT +0000. [14:50:07] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:20] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:23] PROBLEM - wiki.aridia.space - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 24.13, 15.14, 10.53 [14:51:09] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.033 second response time [14:51:43] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Mon 10 Oct 2022 06:13:03 GMT +0000. [14:51:43] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:51:45] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.069 second response time [14:51:53] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [14:52:24] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.775 second response time [14:52:33] PROBLEM - mario.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:17] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.89, 10.54, 11.08 [14:53:41] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [14:53:51] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:14] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:15] PROBLEM - cloudwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:29] PROBLEM - uncyclopediakr.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:36] PROBLEM - wiki.horizonsend.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [14:55:16] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.060 second response time [14:55:22] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:21] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 17.00, 10.07, 6.91 [14:56:24] PROBLEM - mazakwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.31, 10.79, 10.18 [14:56:39] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:58:05] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:58:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.50, 10.02, 7.31 [14:58:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 18.45, 13.34, 11.17 [14:58:47] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:16] PROBLEM - www.programming.red - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:18] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.42, 2.85 [14:59:37] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:43] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:54] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:54] PROBLEM - iol.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:59] PROBLEM - wiki.puucraft.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:02] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:07] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:10] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:00:28] PROBLEM - wiki.potabi.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:37] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39171 bytes in 1.037 second response time [15:01:15] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 3.23, 2.86 [15:01:27] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.031 second response time [15:01:37] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:49] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:03] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.052 second response time [15:02:05] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.338 second response time [15:02:10] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [15:02:15] PROBLEM - mrlove.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:33] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:34] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:45] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:03:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 11.50, 11.82 [15:03:33] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:03:44] PROBLEM - famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:51] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.239 second response time [15:03:52] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:04:09] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:12] PROBLEM - www.obeymewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.13, 10.18, 10.70 [15:05:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.92, 0.99, 0.78 [15:05:45] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:48] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.020 second response time [15:05:50] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:21] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.00, 10.39, 8.27 [15:06:29] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Tue 23 Aug 2022 02:51:23 GMT +0000. [15:06:31] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [15:07:09] PROBLEM - history.estill.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:32] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:07:37] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [15:07:48] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.036 second response time [15:08:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 6.07, 8.71, 7.91 [15:08:30] PROBLEM - dcmultiverse.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.00, 8.05, 9.76 [15:09:04] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:09:05] RECOVERY - wiki.virtualjet.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Sun 25 Sep 2022 21:20:59 GMT +0000. [15:09:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.06, 12.55, 11.91 [15:09:31] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.222 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [15:09:34] PROBLEM - smafallfest.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:34] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sat 01 Oct 2022 04:26:41 GMT +0000. [15:09:43] RECOVERY - arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Sat 17 Sep 2022 21:47:49 GMT +0000. [15:10:10] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:25] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:10:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [15:10:57] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:15] RECOVERY - www.opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [15:11:16] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sun 30 Oct 2022 08:17:32 GMT +0000. [15:11:54] RECOVERY - arcanewaterswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'arcanewaterswiki.org' will expire on Sun 30 Oct 2022 20:54:06 GMT +0000. [15:11:56] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:35] RECOVERY - slupe.fp64.org - LetsEncrypt on sslhost is OK: OK - Certificate 'slupe.fp64.org' will expire on Tue 04 Oct 2022 18:47:51 GMT +0000. [15:12:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 11.83, 10.99 [15:12:51] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.314 second response time [15:12:55] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [15:13:16] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:27] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 21 Sep 2022 09:01:23 GMT +0000. [15:14:30] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:14:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.11, 11.46, 10.96 [15:14:43] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [15:14:43] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [15:14:48] RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Wed 28 Sep 2022 11:34:56 GMT +0000. [15:14:57] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:15:32] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:16:31] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:17:27] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:17:27] RECOVERY - swests.org - LetsEncrypt on sslhost is OK: OK - Certificate 'swests.org' will expire on Thu 20 Oct 2022 03:33:41 GMT +0000. [15:17:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 3.97, 3.43 [15:18:14] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.061 second response time [15:18:35] RECOVERY - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pbptanarchy.tk' will expire on Tue 06 Sep 2022 20:52:59 GMT +0000. [15:18:35] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [15:18:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.73, 9.01, 10.10 [15:18:39] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:19:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.79, 3.41 [15:19:54] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 9110 MB (23% inode=96%); [15:20:04] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sat 17 Sep 2022 22:00:09 GMT +0000. [15:20:42] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.021 second response time [15:20:48] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:58] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [15:22:00] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:22:25] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:27] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Tue 23 Aug 2022 02:37:13 GMT +0000. [15:22:33] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:22:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.62, 2.53, 1.62 [15:22:46] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [15:23:09] RECOVERY - www.proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [15:23:18] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:23:19] RECOVERY - cloudwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cloudwiki.org' will expire on Sun 23 Oct 2022 16:49:57 GMT +0000. [15:23:48] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.171 second response time [15:23:50] RECOVERY - wiki.horizonsend.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.horizonsend.net' will expire on Thu 25 Aug 2022 17:01:32 GMT +0000. [15:24:02] RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:24:36] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.313 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25 [15:24:56] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.054 second response time [15:25:19] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:24] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [15:25:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 10.86, 11.87 [15:25:34] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 5.078 second response time [15:25:51] RECOVERY - mazakwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mazakwiki.org' will expire on Wed 26 Oct 2022 18:25:25 GMT +0000. [15:26:54] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:27:29] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.46, 4.88, 3.85 [15:27:37] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.040 second response time [15:27:48] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:06] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Sat 17 Sep 2022 22:20:19 GMT +0000. [15:28:15] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Thu 27 Oct 2022 15:02:13 GMT +0000. [15:28:23] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:28:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.20, 11.01, 10.26 [15:28:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.95, 1.65 [15:28:50] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Tue 23 Aug 2022 09:31:28 GMT +0000. [15:28:51] RECOVERY - iol.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'iol.wiki' will expire on Thu 27 Oct 2022 14:58:25 GMT +0000. [15:28:51] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [15:28:52] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [15:29:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.86, 12.95, 12.39 [15:29:28] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:29:54] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Mon 10 Oct 2022 06:24:27 GMT +0000. [15:30:17] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Thu 01 Sep 2022 00:43:35 GMT +0000. [15:30:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.49, 10.55, 10.23 [15:30:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.58, 1.55 [15:31:00] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:45] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [15:31:51] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Mon 22 Aug 2022 23:58:24 GMT +0000. [15:32:12] PROBLEM - wiki.fossbots.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:14] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:17] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.61, 8.48, 9.51 [15:33:03] RECOVERY - www.obeymewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.obeymewiki.com' will expire on Thu 01 Sep 2022 00:44:40 GMT +0000. [15:33:03] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [15:33:10] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.77, 3.68 [15:33:39] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:39] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:09] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:13] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:13] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:25] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:34:34] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 23 Aug 2022 00:48:11 GMT +0000. [15:34:56] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:07] PROBLEM - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:14] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:35:38] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.670 second response time [15:36:04] PROBLEM - wikicryptos.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:13] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw121 mw122 mw131 [15:36:41] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:55] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:13] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 8862 MB (23% inode=96%); [15:37:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 2.95, 3.36 [15:37:31] PROBLEM - wiki.recaptime.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:38:07] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.43, 10.61, 8.29 [15:38:29] PROBLEM - webkinzguide.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:54] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:57] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [15:39:37] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:37] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:41] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:39:48] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [15:40:04] PROBLEM - miraheze.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:11] PROBLEM - creativecommons.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:04] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.013 second response time [15:41:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.85, 8.56, 11.84 [15:41:28] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:42] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.109 second response time [15:41:46] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.101 second response time [15:42:05] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.72, 9.75, 8.48 [15:42:55] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [15:42:55] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.318 second response time [15:43:02] PROBLEM - theapplesecretswiki.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:36] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:36] PROBLEM - wiki.recaptime.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.46, 13.42, 10.43 [15:44:51] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:55] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:01] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:45:08] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.174 second response time [15:45:18] PROBLEM - en.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:32] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:33] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.040 second response time [15:45:41] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:42] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.026 second response time [15:46:03] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:03] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:07] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:20] PROBLEM - ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:51] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [15:46:53] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.323 second response time [15:46:57] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:47:18] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.169 second response time [15:47:23] RECOVERY - buildabearwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'buildabearwiki.info' will expire on Fri 16 Sep 2022 17:59:19 GMT +0000. [15:47:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.22, 7.28, 10.02 [15:47:58] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:34] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [15:48:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:24] RECOVERY - wiki.aridia.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Sat 24 Sep 2022 13:17:20 GMT +0000. [15:50:04] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.325 second response time [15:50:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 10.70, 10.30 [15:51:02] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.037 second response time [15:51:29] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.019 second response time [15:52:10] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:52:25] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:29] PROBLEM - www.ethene.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.00, 8.59, 9.57 [15:52:42] PROBLEM - wiki.infomedia.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.35, 1.75, 1.33 [15:53:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.53, 13.67, 11.99 [15:53:58] PROBLEM - commons.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:11] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.043 second response time [15:55:01] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:55:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.68, 11.09, 11.28 [15:55:38] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:55:39] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:56:20] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:26] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:39] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.76, 1.44, 1.32 [15:56:57] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:13] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.021 second response time [15:57:40] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.023 second response time [15:58:10] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3402 bytes in 7.230 second response time [15:58:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.12, 11.58, 10.24 [15:58:42] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Tue 23 Aug 2022 03:02:35 GMT +0000. [15:58:46] [02mw-config] 07Universal-Omega closed pull request 03#4849: T9629: make editextendedconfirmedprotected and edittemplateprotected available in theredpionnerwiki - 13https://github.com/miraheze/mw-config/pull/4849 [15:58:46] [url] Page not found · GitHub · GitHub | github.com [15:58:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/11cbd0d07a9c...ca4d083be96d [15:58:48] [url] Comparing 11cbd0d07a9c...ca4d083be96d · miraheze/mw-config · GitHub | github.com [15:58:49] [02miraheze/mw-config] 07ugochimobi 03ca4d083 - T9629: make editextendedconfirmedprotected and edittemplateprotected available in theredpionnerwiki (#4849) [15:58:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.85, 3.52 [15:59:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.29, 8.07, 9.93 [15:59:39] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:44] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:59] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Sat 01 Oct 2022 16:16:26 GMT +0000. [16:00:03] miraheze/mw-config - Universal-Omega the build passed. [16:00:15] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ca4d083be96d...1302e044b497 [16:00:16] [url] Comparing ca4d083be96d...1302e044b497 · miraheze/mw-config · GitHub | github.com [16:00:17] [02miraheze/mw-config] 07Universal-Omega 031302e04 - Remove theredpionnerwiki from ManageWikiPermissionsAdditionalRights [16:00:28] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:38] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:00:49] PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:53] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:00:56] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:25] miraheze/mw-config - Universal-Omega the build passed. [16:01:37] RECOVERY - wiki.fossbots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fossbots.org' will expire on Sun 16 Oct 2022 13:11:42 GMT +0000. [16:01:49] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sun 02 Oct 2022 10:22:40 GMT +0000. [16:01:58] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:26] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [16:02:42] PROBLEM - correypedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:42] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.039 second response time [16:02:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.24, 3.34 [16:02:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [16:02:55] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Tue 23 Aug 2022 01:21:14 GMT +0000. [16:02:57] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.041 second response time [16:02:58] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sat 17 Sep 2022 18:42:17 GMT +0000. [16:03:58] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Tue 23 Aug 2022 01:40:20 GMT +0000. [16:04:13] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 5.121 second response time [16:04:48] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Wed 24 Aug 2022 03:29:39 GMT +0000. [16:04:49] RECOVERY - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'mcdev.grantlmul.xyz' will expire on Fri 14 Oct 2022 16:18:23 GMT +0000. [16:05:05] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:15:36 GMT +0000. [16:05:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 23.82, 15.28, 12.10 [16:05:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.014 second response time [16:05:56] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:25] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:32] PROBLEM - roblox-wiki.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:33] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:06:36] RECOVERY - wiki.recaptime.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.eu.org' will expire on Mon 03 Oct 2022 22:23:06 GMT +0000. [16:06:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 4.97, 10.04, 10.70 [16:07:21] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:26] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:41] PROBLEM - civclassic.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:45] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Wed 24 Aug 2022 05:14:40 GMT +0000. [16:08:29] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.024 second response time [16:08:34] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:08:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.44, 8.29, 9.98 [16:08:39] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.038 second response time [16:09:21] !log [@mwtask141] starting deploy of {'config': True} to all [16:09:26] RECOVERY - threedomwiki.pcast.site - LetsEncrypt on sslhost is OK: OK - Certificate 'threedomwiki.pcast.site' will expire on Mon 22 Aug 2022 22:02:33 GMT +0000. [16:09:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:37] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.772 second response time [16:10:51] RECOVERY - biblestrength.net - LetsEncrypt on sslhost is OK: OK - Certificate 'biblestrength.net' will expire on Wed 07 Sep 2022 18:16:09 GMT +0000. [16:11:13] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.089 second response time [16:11:18] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.058 second response time [16:11:19] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 24 Aug 2022 02:59:35 GMT +0000. [16:11:30] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.353 second response time [16:11:33] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [16:11:35] PROBLEM - yokaiwatch.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:58] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw141.miraheze.org [16:12:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:13] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29520 bytes in 0.386 second response time [16:12:24] RECOVERY - theapplesecretswiki.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'theapplesecretswiki.ml' will expire on Mon 17 Oct 2022 19:49:07 GMT +0000. [16:12:45] RECOVERY - wiki.recaptime.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.tk' will expire on Wed 26 Oct 2022 23:10:27 GMT +0000. [16:14:10] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [16:14:33] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [16:14:42] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [16:14:43] RECOVERY - en.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiretet.ga' will expire on Sun 02 Oct 2022 05:37:01 GMT +0000. [16:14:43] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [16:15:12] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 12.00, 8.16, 5.71 [16:15:16] RECOVERY - ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ballmedia.org' will expire on Mon 10 Oct 2022 05:56:45 GMT +0000. [16:15:25] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:27] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:11] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.13, 7.39, 5.73 [16:17:49] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:18:32] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [16:18:45] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:45] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.018 second response time [16:21:34] RECOVERY - www.ethene.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ethene.wiki' will expire on Sat 24 Sep 2022 13:58:45 GMT +0000. [16:21:41] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 3% [16:21:43] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:58] RECOVERY - wiki.infomedia.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.infomedia.co.uk' will expire on Fri 21 Oct 2022 15:33:32 GMT +0000. [16:22:25] RECOVERY - en.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.pornwiki.org' will expire on Sat 17 Sep 2022 22:00:09 GMT +0000. [16:22:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.25, 9.97, 9.12 [16:22:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.04, 2.42, 1.85 [16:22:56] RECOVERY - commons.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.ballmedia.org' will expire on Mon 10 Oct 2022 06:16:04 GMT +0000. [16:23:56] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.048 second response time [16:24:16] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:24:19] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:24:31] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 8.95, 8.85 [16:24:37] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:40] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:51] PROBLEM - wiki.wisrail.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:34] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:06] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:26:15] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:17] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [16:26:17] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:22] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 3 % full [16:26:30] !log [@test131] starting deploy of {'config': True} to all [16:26:31] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:26:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.74, 1.71 [16:26:43] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.355 second response time [16:27:01] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [16:27:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:30] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:43] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.88, 9.44, 6.67 [16:27:45] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [16:27:45] [url] Page not found · GitHub · GitHub | github.com [16:27:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0fafcdf0776a...658367c06d97 [16:27:47] [url] Comparing 0fafcdf0776a...658367c06d97 · miraheze/mw-config · GitHub | github.com [16:27:48] [02miraheze/mw-config] 07Universal-Omega 03658367c - Update MirahezeFunctions.php [16:27:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:28:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:28:06] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:24] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:28:39] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.48, 1.61 [16:29:00] miraheze/mw-config - Universal-Omega the build passed. [16:29:12] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [16:29:21] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.015 second response time [16:29:28] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:31] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Thu 22 Sep 2022 04:19:34 GMT +0000. [16:29:31] RECOVERY - wiki.puucraft.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.puucraft.net' will expire on Wed 26 Oct 2022 16:57:15 GMT +0000. [16:29:36] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:29:39] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:42] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.50, 7.42, 6.26 [16:29:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 4.16, 3.61 [16:29:42] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:58] PROBLEM - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:38] RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Sun 30 Oct 2022 15:35:11 GMT +0000. [16:31:11] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.083 second response time [16:31:13] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:28] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:31:44] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.271 second response time [16:31:45] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:31:46] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:32:00] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Tue 23 Aug 2022 08:44:56 GMT +0000. [16:32:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [16:32:01] [url] Page not found · GitHub · GitHub | github.com [16:32:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/658367c06d97...4eb71ff618f2 [16:32:03] [url] Comparing 658367c06d97...4eb71ff618f2 · miraheze/mw-config · GitHub | github.com [16:32:04] [02miraheze/mw-config] 07Universal-Omega 034eb71ff - Update MirahezeFunctions.php [16:32:17] PROBLEM - wiki.cjgh.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:31] PROBLEM - wiki.site97.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:36] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Wed 12 Oct 2022 13:11:41 GMT +0000. [16:33:10] miraheze/mw-config - Universal-Omega the build passed. [16:33:15] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.18, 9.93, 11.92 [16:33:28] PROBLEM - wiki.fossbots.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.97, 3.67 [16:33:42] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.021 second response time [16:34:09] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:14] PROBLEM - famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:18] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:18] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:25] RECOVERY - wikicryptos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicryptos.org' will expire on Wed 24 Aug 2022 05:45:16 GMT +0000. [16:34:38] PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:42] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:43] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 4.45, 3.87 [16:35:28] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:39] PROBLEM - pt.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:39] PROBLEM - wiki.auraxis.co - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:54] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:25] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.66, 9.47, 8.26 [16:36:40] RECOVERY - webkinzguide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'webkinzguide.com' will expire on Mon 29 Aug 2022 18:06:45 GMT +0000. [16:36:54] PROBLEM - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:58] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.078 second response time [16:37:02] PROBLEM - fr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:21] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:37:22] PROBLEM - wiki.otir.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:24] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:37:26] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:12] PROBLEM - largedu.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:22] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.57, 7.35, 7.63 [16:38:38] PROBLEM - reviews.thejoshmeister.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:57] PROBLEM - en.omniversalis.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:00] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 7.130 second response time [16:39:03] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:06] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:20] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.031 second response time [16:39:21] RECOVERY - miraheze.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.cf' will expire on Wed 02 Nov 2022 19:48:19 GMT +0000. [16:39:33] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:39:37] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:41] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.451 second response time [16:39:49] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:39:57] PROBLEM - orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:59] PROBLEM - www.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:08] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [16:40:10] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:33] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [16:41:06] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:41:36] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.044 second response time [16:41:49] PROBLEM - arru.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:12] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:20] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:42:45] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.022 second response time [16:42:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:43:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4eb71ff618f2...9be12b85f7e1 [16:43:10] [url] Comparing 4eb71ff618f2...9be12b85f7e1 · miraheze/mw-config · GitHub | github.com [16:43:11] [02miraheze/mw-config] 07Universal-Omega 039be12b8 - Update MirahezeFunctions.php [16:43:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [16:43:13] [url] Page not found · GitHub · GitHub | github.com [16:43:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.62, 3.75 [16:43:25] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:41] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:16] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:17] PROBLEM - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:18] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:21] PROBLEM - wiki.recaptime.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:22] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.246 second response time [16:44:30] miraheze/mw-config - Universal-Omega the build passed. [16:44:52] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address tl.awiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:44:57] PROBLEM - source.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:13] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Sat 17 Sep 2022 22:08:07 GMT +0000. [16:45:29] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:46:12] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:25] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.186 second response time [16:46:30] PROBLEM - royal-wiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:42] PROBLEM - sharqedia.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:10] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:53] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:56] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:02] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.016 second response time [16:48:24] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:39] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [16:48:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:41] PROBLEM - phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:05] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:07] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.04, 9.23, 7.23 [16:49:11] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.022 second response time [16:49:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.14, 2.95, 3.37 [16:49:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 23.38, 14.85, 12.39 [16:50:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9be12b85f7e1...ed78c0489f81 [16:50:09] [url] Comparing 9be12b85f7e1...ed78c0489f81 · miraheze/mw-config · GitHub | github.com [16:50:10] [02miraheze/mw-config] 07Universal-Omega 03ed78c04 - Update MirahezeFunctions.php [16:50:12] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [16:50:12] [url] Page not found · GitHub · GitHub | github.com [16:50:27] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 2 % full [16:50:31] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:07] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.23, 8.00, 7.02 [16:51:36] miraheze/mw-config - Universal-Omega the build passed. [16:51:50] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:52:20] PROBLEM - www.electrowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [16:53:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.39, 3.46 [16:53:51] PROBLEM - podpedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:19] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [16:54:34] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Fri 02 Sep 2022 12:34:13 GMT +0000. [16:54:38] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Wed 24 Aug 2022 04:03:58 GMT +0000. [16:54:46] RECOVERY - wiki.wisrail.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wisrail.de' will expire on Tue 06 Sep 2022 21:01:07 GMT +0000. [16:55:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.93, 3.80, 3.58 [16:55:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.95, 11.08, 11.87 [16:55:39] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Fri 07 Oct 2022 11:49:47 GMT +0000. [16:55:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=alcriswiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/alcriswiki-20220805164746.xml (START) [16:55:55] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:02] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thegigaversewiki --no-updates --username-prefix=wikia:thegigaverse /mnt/mediawiki-static/metawiki/ImportDump/thegigaversewiki-20220805124205.xml (START) [16:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:10] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:58:24] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [16:58:28] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Mon 22 Aug 2022 23:35:07 GMT +0000. [16:58:58] RECOVERY - elsterclopedie.marithaime.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'elsterclopedie.marithaime.nl' will expire on Sun 30 Oct 2022 20:08:40 GMT +0000. [16:59:00] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:59:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=alcriswiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/alcriswiki-20220805164746.xml (END - exit=0) [16:59:05] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=alcriswiki (START) [16:59:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:59:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:49] PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:01] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 24 Aug 2022 04:38:47 GMT +0000. [17:01:06] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=alcriswiki (END - exit=0) [17:01:07] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=alcriswiki --active --update (END - exit=0) [17:01:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.92, 3.84 [17:01:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.11, 12.40, 11.88 [17:01:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:01:40] RECOVERY - wiki.site97.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.site97.fr' will expire on Tue 18 Oct 2022 15:25:34 GMT +0000. [17:01:51] RECOVERY - wiki.cjgh.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cjgh.xyz' will expire on Mon 03 Oct 2022 20:57:39 GMT +0000. [17:01:54] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Tue 23 Aug 2022 09:01:11 GMT +0000. [17:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:01:58] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [17:01:58] [url] Page not found · GitHub · GitHub | github.com [17:01:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ed78c0489f81...bc566d5f2e9b [17:02:00] [url] Comparing ed78c0489f81...bc566d5f2e9b · miraheze/mw-config · GitHub | github.com [17:02:01] [02miraheze/mw-config] 07Universal-Omega 03bc566d5 - Update MirahezeFunctions.php [17:02:05] RECOVERY - correypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'correypedia.org' will expire on Thu 01 Sep 2022 00:41:01 GMT +0000. [17:02:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.78, 1.52 [17:02:54] RECOVERY - wiki.fossbots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fossbots.org' will expire on Sun 16 Oct 2022 13:11:42 GMT +0000. [17:02:55] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Tue 23 Aug 2022 01:21:14 GMT +0000. [17:03:03] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Wed 24 Aug 2022 02:28:53 GMT +0000. [17:03:06] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.053 second response time [17:03:07] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.258 second response time [17:03:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bc566d5f2e9b...82d8a85bd341 [17:03:11] [url] Comparing bc566d5f2e9b...82d8a85bd341 · miraheze/mw-config · GitHub | github.com [17:03:12] [02miraheze/mw-config] 07Universal-Omega 0382d8a85 - Update .phpcs.xml [17:03:12] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [17:03:13] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Tue 23 Aug 2022 08:56:40 GMT +0000. [17:03:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [17:03:14] [url] Page not found · GitHub · GitHub | github.com [17:03:22] miraheze/mw-config - Universal-Omega the build passed. [17:03:32] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:33] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [17:04:02] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sat 17 Sep 2022 18:42:17 GMT +0000. [17:04:19] RECOVERY - wiki.thedev.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thedev.gq' will expire on Fri 02 Sep 2022 12:36:20 GMT +0000. [17:04:24] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.015 second response time [17:04:24] RECOVERY - wiki.auraxis.co - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.auraxis.co' will expire on Fri 21 Oct 2022 06:34:16 GMT +0000. [17:04:26] miraheze/mw-config - Universal-Omega the build has errored. [17:04:30] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Tue 23 Aug 2022 01:40:20 GMT +0000. [17:04:34] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [17:04:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.48, 1.68, 1.52 [17:04:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thegigaversewiki --no-updates --username-prefix=wikia:thegigaverse /mnt/mediawiki-static/metawiki/ImportDump/thegigaversewiki-20220805124205.xml (END - exit=0) [17:04:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=thegigaversewiki (START) [17:04:52] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Mon 22 Aug 2022 21:48:31 GMT +0000. [17:05:02] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.67, 2.72, 3.36 [17:05:24] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:33] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:34] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/82d8a85bd341...8d91dbdd0921 [17:05:36] [url] Comparing 82d8a85bd341...8d91dbdd0921 · miraheze/mw-config · GitHub | github.com [17:05:37] [02miraheze/mw-config] 07Universal-Omega 038d91dbd - Update MirahezeFunctions.php [17:05:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [17:05:39] [url] Page not found · GitHub · GitHub | github.com [17:05:41] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.058 second response time [17:05:45] RECOVERY - roblox-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'roblox-wiki.tk' will expire on Thu 27 Oct 2022 15:10:45 GMT +0000. [17:05:49] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:07] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:09] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:06:26] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:31] RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:06:35] RECOVERY - fr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fr.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:15:36 GMT +0000. [17:06:36] RECOVERY - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'mcdev.grantlmul.xyz' will expire on Fri 14 Oct 2022 16:18:23 GMT +0000. [17:06:44] RECOVERY - smafallfest.org - LetsEncrypt on sslhost is OK: OK - Certificate 'smafallfest.org' will expire on Wed 26 Oct 2022 18:44:23 GMT +0000. [17:06:46] miraheze/mw-config - Universal-Omega the build passed. [17:07:11] RECOVERY - civclassic.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:07:26] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 1 % full [17:07:27] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:07:36] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.399 second response time [17:07:40] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.031 second response time [17:07:42] RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Sun 30 Oct 2022 14:46:40 GMT +0000. [17:07:49] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:52] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=thegigaversewiki (END - exit=0) [17:07:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=thegigaversewiki --active --update (END - exit=0) [17:07:54] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Wed 24 Aug 2022 05:14:40 GMT +0000. [17:07:59] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Wed 26 Oct 2022 20:54:04 GMT +0000. [17:07:59] [02miraheze/mw-config] 07Universal-Omega pushed 033 commits to 03Universal-Omega-patch-1 [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/8d91dbdd0921...aabb6bf0177e [17:08:00] [url] Comparing 8d91dbdd0921...aabb6bf0177e · miraheze/mw-config · GitHub | github.com [17:08:01] [02miraheze/mw-config] 07Universal-Omega 03aabb6bf - Merge branch 'master' into Universal-Omega-patch-1 [17:08:02] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [17:08:03] [url] Page not found · GitHub · GitHub | github.com [17:08:06] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.312 second response time [17:08:10] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [17:08:12] RECOVERY - reviews.thejoshmeister.com - LetsEncrypt on sslhost is OK: OK - Certificate 'reviews.thejoshmeister.com' will expire on Thu 29 Sep 2022 08:42:34 GMT +0000. [17:08:33] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [17:08:35] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/aabb6bf0177e...3b0c694aee79 [17:08:35] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [17:08:36] [url] Comparing aabb6bf0177e...3b0c694aee79 · miraheze/mw-config · GitHub | github.com [17:08:37] [02miraheze/mw-config] 07github-actions 033b0c694 - CI: lint code to MediaWiki standards [17:08:38] [02mw-config] 07github-actions[bot] synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [17:08:38] [url] Page not found · GitHub · GitHub | github.com [17:08:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.76, 1.58 [17:08:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw121 mw141 [17:09:10] miraheze/mw-config - Universal-Omega the build passed. [17:09:12] RECOVERY - yokaiwatch.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:09:22] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw141 [17:09:29] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 0.046 second response time [17:09:49] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:52] RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Thu 27 Oct 2022 05:20:13 GMT +0000. [17:10:40] RECOVERY - arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Sat 17 Sep 2022 21:47:49 GMT +0000. [17:10:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.39, 1.47 [17:11:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.09, 2.47, 3.84 [17:11:23] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [17:11:45] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:02] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [17:12:05] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:12:14] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [17:12:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:59] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.128 second response time [17:13:09] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:13:22] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:13:25] RECOVERY - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.phgalaxy.eu.org' will expire on Mon 10 Oct 2022 06:42:26 GMT +0000. [17:13:27] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Mon 22 Aug 2022 23:01:19 GMT +0000. [17:13:29] RECOVERY - wiki.recaptime.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.tk' will expire on Wed 26 Oct 2022 23:10:27 GMT +0000. [17:13:37] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:41] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: add function doc comments - 13https://github.com/miraheze/mw-config/pull/4848 [17:13:41] [url] Page not found · GitHub · GitHub | github.com [17:13:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b0c694aee79...605fa95ab050 [17:13:43] [url] Comparing 3b0c694aee79...605fa95ab050 · miraheze/mw-config · GitHub | github.com [17:13:44] [02miraheze/mw-config] 07Universal-Omega 03605fa95 - Update .phpcs.xml [17:13:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.52, 3.21, 2.19 [17:13:54] RECOVERY - tl.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'tl.awiki.org' will expire on Tue 23 Aug 2022 07:53:50 GMT +0000. [17:13:58] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [17:13:58] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.307 second response time [17:14:00] [02mw-config] 07Universal-Omega edited pull request 03#4848: MirahezeFunctions: enable phpcs doc comments rules - 13https://github.com/miraheze/mw-config/pull/4848 [17:14:00] [url] Page not found · GitHub · GitHub | github.com [17:14:10] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:15] RECOVERY - arcanewaterswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'arcanewaterswiki.org' will expire on Sun 30 Oct 2022 20:54:06 GMT +0000. [17:14:18] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 7.127 second response time [17:14:36] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 52% [17:14:48] miraheze/mw-config - Universal-Omega the build passed. [17:14:58] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 17 Sep 2022 21:39:52 GMT +0000. [17:15:06] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.042 second response time [17:15:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 2.02, 3.32 [17:15:23] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [17:15:41] RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Wed 28 Sep 2022 11:34:56 GMT +0000. [17:15:46] RECOVERY - sharqedia.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'sharqedia.ml' will expire on Mon 03 Oct 2022 13:40:45 GMT +0000. [17:15:55] PROBLEM - en.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:56] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:43] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 6.677 second response time [17:16:51] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Tue 23 Aug 2022 07:40:05 GMT +0000. [17:17:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.86, 9.26, 11.50 [17:17:37] PROBLEM - wiki.projectdiablo2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:44] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [17:17:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.85, 1.80, 1.83 [17:17:51] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.051 second response time [17:18:04] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.044 second response time [17:18:07] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.314 second response time [17:18:22] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [17:18:31] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.363 second response time [17:18:32] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.747 second response time [17:18:34] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 4% [17:18:34] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 14 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [17:18:38] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [17:18:45] PROBLEM - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:11] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.24, 3.62, 3.61 [17:19:23] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [17:19:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 9.86, 11.38 [17:19:58] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:05] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:20:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [17:21:33] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.031 second response time [17:21:52] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:56] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [17:22:05] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39170 bytes in 0.034 second response time [17:22:17] PROBLEM - ndg.nenawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:20] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:23] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:22:49] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [17:23:24] PROBLEM - rarewiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:32] PROBLEM - wiki.ieg123.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:44] PROBLEM - miraheze.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:55] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:24:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:24:12] PROBLEM - www.proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:14] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:33] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.069 second response time [17:24:34] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.055 second response time [17:24:37] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:24:43] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:25:38] PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.42, 1.52, 1.70 [17:26:37] PROBLEM - wikien.wildterra2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:05] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:06] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:07] PROBLEM - datawiki.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:19] PROBLEM - electrowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 4.77, 9.65, 11.52 [17:27:49] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:57] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:17] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:28:49] PROBLEM - animatedfeet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:35] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:36] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:42] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:44] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.061 second response time [17:30:04] PROBLEM - steamdecklinux.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:33] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:57] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 10.40, 11.26 [17:31:41] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:31:44] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.850 second response time [17:32:23] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.60, 11.32, 11.53 [17:33:34] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:33:37] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:46] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:50] PROBLEM - wiki.cjgh.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:20] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.041 second response time [17:34:24] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:00] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:00] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:18] PROBLEM - famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:28] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Fri 26 Aug 2022 04:20:05 GMT +0000. [17:35:31] PROBLEM - www.zetapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:35:31] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:36:05] PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:31] PROBLEM - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:36:58] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 13 day(s) (Fri 19 Aug 2022 17:20:05 GMT +0000). [17:37:18] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3403 bytes in 7.217 second response time [17:37:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.33, 10.87, 8.16 [17:37:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.92, 7.65, 9.95 [17:37:29] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.019 second response time [17:38:07] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sat 01 Oct 2022 04:26:41 GMT +0000. [17:38:14] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [17:38:51] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 2.088 second response time [17:39:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 4.17, 3.40 [17:40:33] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:41:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 21.02, 12.93, 11.38 [17:41:46] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.204 second response time [17:41:57] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:11] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [17:42:29] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 1.042 second response time [17:43:02] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:43:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.38, 9.75, 8.94 [17:43:06] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.84, 3.45 [17:43:26] PROBLEM - wiki.iurevar.world - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:31] PROBLEM - slupe.fp64.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:57] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:27] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 21 Sep 2022 09:01:23 GMT +0000. [17:45:05] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.054 second response time [17:45:19] RECOVERY - en.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiretet.ga' will expire on Sun 02 Oct 2022 05:37:01 GMT +0000. [17:45:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.70, 3.13, 3.24 [17:46:09] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:16] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Wed 24 Aug 2022 04:30:45 GMT +0000. [17:46:18] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:19] RECOVERY - wiki.projectdiablo2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.projectdiablo2.com' will expire on Tue 11 Oct 2022 03:46:46 GMT +0000. [17:47:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [17:47:11] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [17:47:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.84, 1.50 [17:47:29] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:47:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 2.08, 1.63 [17:48:03] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Wed 12 Oct 2022 13:33:40 GMT +0000. [17:48:30] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [17:48:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:54] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.11, 10.90, 11.50 [17:49:32] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:50:54] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.044 second response time [17:51:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 19.24, 13.27, 12.22 [17:51:34] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.93, 1.69 [17:51:47] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:51:53] RECOVERY - ndg.nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Tue 23 Aug 2022 01:35:44 GMT +0000. [17:52:01] RECOVERY - www.electrowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electrowiki.org' will expire on Sat 17 Sep 2022 19:15:15 GMT +0000. [17:52:02] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:22] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [17:52:23] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.287 second response time [17:52:51] RECOVERY - rarewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewiki.org' will expire on Wed 28 Sep 2022 05:06:19 GMT +0000. [17:53:02] PROBLEM - wiki.minebox.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:03] RECOVERY - wiki.ieg123.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ieg123.com' will expire on Fri 14 Oct 2022 17:56:38 GMT +0000. [17:53:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.61, 1.51 [17:53:19] RECOVERY - www.proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [17:53:28] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:33] RECOVERY - miraheze.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.tk' will expire on Tue 23 Aug 2022 07:48:08 GMT +0000. [17:53:59] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 8655 MB (22% inode=96%); [17:54:08] RECOVERY - grk.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Sat 17 Sep 2022 21:13:49 GMT +0000. [17:54:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 23.63, 14.16, 10.25 [17:54:44] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:55:02] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:19] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Wed 24 Aug 2022 01:30:15 GMT +0000. [17:55:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 1.87, 1.70 [17:55:38] RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [17:56:14] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [17:56:22] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.010 second response time [17:56:24] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Wed 24 Aug 2022 04:25:02 GMT +0000. [17:56:29] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [17:56:39] RECOVERY - electrowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electrowiki.org' will expire on Sat 17 Sep 2022 19:15:15 GMT +0000. [17:56:45] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.050 second response time [17:56:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.98, 1.70 [17:57:01] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:16] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:57:17] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:28] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:57:34] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.89, 1.73 [17:58:13] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:29:43 GMT +0000. [17:58:13] RECOVERY - animatedfeet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [17:58:28] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Mon 22 Aug 2022 23:35:07 GMT +0000. [17:58:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.37, 10.36, 9.63 [17:58:46] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Mon 22 Aug 2022 22:32:53 GMT +0000. [17:58:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.64, 1.61 [17:58:57] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:00] [02miraheze/TSPortal] 07OwenBaines pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/TSPortal/compare/2ce35447995c...e9ea6bc7e4d1 [17:59:00] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [17:59:00] [url] Comparing 2ce35447995c...e9ea6bc7e4d1 · miraheze/TSPortal · GitHub | github.com [17:59:01] [02miraheze/TSPortal] 07OwenBaines 03e9ea6bc - [Release] API: Use correct name for DPA subject [17:59:02] [02miraheze/TSPortal] 07OwenBaines tagged 03e9ea6bc as 03v7 13https://github.com/miraheze/TSPortal/commit/e9ea6bc7e4d12cd45e958c30bfa3a70322ee728c [17:59:04] [02TSPortal] 07OwenBaines created tag 03v7 - 13https://github.com/miraheze/TSPortal [17:59:05] [url] Page not found · GitHub · GitHub | github.com [17:59:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.12, 3.55, 3.22 [17:59:22] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Thu 01 Sep 2022 00:43:35 GMT +0000. [17:59:37] PROBLEM - roll-of-arms.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:46] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.196 second response time [18:00:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.52, 8.97, 9.20 [18:01:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.46, 3.22, 3.14 [18:01:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.97, 1.61, 1.66 [18:01:59] PROBLEM - cp21 Stunnel HTTP for mwtask141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:02:02] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.96, 2.02, 1.76 [18:02:49] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:08] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:03:11] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 8.090 second response time [18:03:20] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:24] RECOVERY - wiki.cjgh.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cjgh.xyz' will expire on Mon 03 Oct 2022 20:57:39 GMT +0000. [18:03:25] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Tue 23 Aug 2022 09:28:23 GMT +0000. [18:03:59] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 2.747 second response time [18:04:01] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:04:21] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:30] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:31] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Mon 22 Aug 2022 21:26:08 GMT +0000. [18:04:38] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [18:05:06] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [18:05:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.19, 1.74, 1.73 [18:05:46] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:05:59] RECOVERY - wiki.creaturathegame.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Mon 22 Aug 2022 20:43:25 GMT +0000. [18:06:08] RECOVERY - wiki.digitalefitheid.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitalefitheid.nl' will expire on Wed 21 Sep 2022 08:49:04 GMT +0000. [18:06:24] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.449 second response time [18:06:24] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:06:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 3.266 second response time [18:07:11] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:07:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.32, 1.59, 1.67 [18:07:43] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [18:07:47] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3404 bytes in 7.103 second response time [18:07:51] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [18:08:48] PROBLEM - www.istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:08] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [18:09:32] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/92d1da6e722a...56c79fda281f [18:09:32] [url] Comparing 92d1da6e722a...56c79fda281f · miraheze/puppet · GitHub | github.com [18:09:33] [02miraheze/puppet] 07JohnFLewis 0356c79fd - reports: v7 [18:10:24] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.016 second response time [18:10:30] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:58] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:11:09] RECOVERY - feathercare.net - LetsEncrypt on sslhost is OK: OK - Certificate 'feathercare.net' will expire on Sat 27 Aug 2022 14:00:44 GMT +0000. [18:11:29] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.15, 1.89, 1.78 [18:11:41] PROBLEM - smashbroswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:45] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.048 second response time [18:11:54] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.016 second response time [18:12:19] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:35] RECOVERY - slupe.fp64.org - LetsEncrypt on sslhost is OK: OK - Certificate 'slupe.fp64.org' will expire on Tue 04 Oct 2022 18:47:51 GMT +0000. [18:12:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.99, 1.87 [18:12:51] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:04] PROBLEM - www.connorjwat.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:18] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:32] PROBLEM - wikimas.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/afba9aee011b [18:14:31] [02miraheze/CreateWiki] 07Universal-Omega 03afba9ae - Don't connect to a database on a cluster [18:14:33] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/CreateWiki [18:14:33] [url] Page not found · GitHub · GitHub | github.com [18:14:34] [02CreateWiki] 07Universal-Omega opened pull request 03#332: Don't connect to a database on a cluster - 13https://github.com/miraheze/CreateWiki/pull/332 [18:14:35] ... [18:14:40] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:14:46] RECOVERY - cp21 Stunnel HTTP for mwtask141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15942 bytes in 0.017 second response time [18:15:17] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [18:15:18] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:19] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [18:15:27] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.60, 1.70 [18:15:44] PROBLEM - www.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:16:19] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.184 second response time [18:16:21] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:29] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:29] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:32] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [18:16:48] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.430 second response time [18:17:03] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:07] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 04 Oct 2022 18:56:56 GMT +0000. [18:18:18] PROBLEM - otcg.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:32] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 13 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [18:18:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.64, 1.20, 1.56 [18:18:53] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:03] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:19:09] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:19:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:25] PROBLEM - intp.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:57] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:01] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:33] PROBLEM - itora.bunnies1230.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:55] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:22:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:23] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:35] PROBLEM - miraheze.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:39] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 1.75, 1.71 [18:23:00] PROBLEM - allthetropes.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:01] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:23:29] miraheze/CreateWiki - Universal-Omega the build passed. [18:23:57] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:22] PROBLEM - avas.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:39] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:24:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.63, 1.66 [18:24:49] PROBLEM - www.proficientprofit.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:01] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:25:02] PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:18] PROBLEM - wiki.sketch.town - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:22] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:32] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Wed 24 Aug 2022 01:38:48 GMT +0000. [18:26:07] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.394 second response time [18:26:19] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.355 second response time [18:26:30] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [18:26:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.17, 1.66, 1.28 [18:26:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 2.11, 1.84 [18:26:51] PROBLEM - www.wikiretet.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:10] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.017 second response time [18:27:15] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:27:19] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:27:23] PROBLEM - commonwealth.0um.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:25] PROBLEM - www.puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:50] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:06] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [18:28:11] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.039 second response time [18:28:28] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw122 [18:28:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:36] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:38] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [18:28:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.72, 1.73 [18:28:50] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.03, 12.64, 10.07 [18:29:03] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:17] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:29:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 5.02, 3.82 [18:29:39] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:02] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_TSPortal] [18:30:05] PROBLEM - iceria.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.254 second response time [18:30:24] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 17 Sep 2022 18:34:21 GMT +0000. [18:30:28] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [18:30:36] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29513 bytes in 6.322 second response time [18:30:40] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.512 second response time [18:30:42] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.46, 1.69, 1.39 [18:30:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.57, 1.68 [18:31:16] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 1.095 second response time [18:31:32] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:54] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.215 second response time [18:32:07] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/afba9aee011b...2de1882ab424 [18:32:08] [url] Comparing afba9aee011b...2de1882ab424 · miraheze/CreateWiki · GitHub | github.com [18:32:09] [02miraheze/CreateWiki] 07Universal-Omega 032de1882 - Update WikiManager.php [18:32:10] PROBLEM - cp21 ferm_active on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#332: Don't connect to a database on a cluster - 13https://github.com/miraheze/CreateWiki/pull/332 [18:32:11] [url] Page not found · GitHub · GitHub | github.com [18:32:19] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:32:27] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.95, 9.02, 7.03 [18:32:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [18:32:44] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:32:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.12, 10.60, 10.00 [18:33:10] PROBLEM - en.clockupwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 4.00, 3.65 [18:33:45] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:34:08] RECOVERY - cp21 ferm_active on cp21 is OK: OK ferm input default policy is set [18:34:09] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:24] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.97, 9.64, 7.53 [18:34:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.62, 9.20, 9.55 [18:34:44] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.076 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,51.195.220.68 [18:35:35] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:35:44] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:45] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:55] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.070 second response time [18:36:16] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.059 second response time [18:36:18] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:25] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:37:07] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:09] PROBLEM - history.estill.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:09] PROBLEM - wikipanel.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 2.85, 3.25 [18:37:38] PROBLEM - cp20 Stunnel HTTP for mon141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:37:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:38:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8942e796967b [18:38:07] [02miraheze/mw-config] 07Universal-Omega 038942e79 - Database: add parsercache cluster to externalLoads [18:38:08] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [18:38:09] [url] Page not found · GitHub · GitHub | github.com [18:38:10] [02mw-config] 07Universal-Omega opened pull request 03#4850: Database: add parsercache cluster to externalLoads - 13https://github.com/miraheze/mw-config/pull/4850 [18:38:10] [url] Page not found · GitHub · GitHub | github.com [18:38:29] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.021 second response time [18:38:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 21.33, 13.76, 11.14 [18:38:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.24, 1.89, 1.57 [18:39:21] miraheze/mw-config - Universal-Omega the build passed. [18:39:25] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:50] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:39:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.07, 0.16, 0.25 [18:39:51] miraheze/CreateWiki - Universal-Omega the build has errored. [18:39:58] RECOVERY - cp20 Stunnel HTTP for mon141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 5.134 second response time [18:39:59] PROBLEM - with.cpt-ra.bid - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:07] PROBLEM - wiki.mostly.vet - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [18:40:42] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.58, 1.79, 1.57 [18:40:58] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:18] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:31] RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Sat 22 Oct 2022 05:43:00 GMT +0000. [18:41:33] RECOVERY - wiki.iurevar.world - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.iurevar.world' will expire on Wed 21 Sep 2022 08:53:59 GMT +0000. [18:41:38] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.215 second response time [18:41:39] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:42] RECOVERY - it.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'it.famepedia.org' will expire on Sat 08 Oct 2022 06:08:33 GMT +0000. [18:42:37] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:42] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.19, 1.59, 1.52 [18:42:43] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 24 Aug 2022 16:50:46 GMT +0000. [18:43:19] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.079 second response time [18:43:42] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.020 second response time [18:44:07] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:29] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:32] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:49] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:45:03] RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Sat 17 Sep 2022 20:10:50 GMT +0000. [18:45:08] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.469 second response time [18:45:08] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.407 second response time [18:45:08] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Tue 23 Aug 2022 08:39:11 GMT +0000. [18:45:14] RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Sun 30 Oct 2022 08:12:03 GMT +0000. [18:45:16] RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Sat 17 Sep 2022 18:24:18 GMT +0000. [18:45:37] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Mon 05 Sep 2022 00:02:52 GMT +0000. [18:45:51] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Wed 24 Aug 2022 05:33:46 GMT +0000. [18:46:02] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.396 second response time [18:46:17] PROBLEM - cp20 Stunnel HTTP for puppet141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:46:27] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.348 second response time [18:46:31] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.428 second response time [18:46:33] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 11.90, 11.79 [18:46:44] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:46] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:47] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 24 Aug 2022 05:02:45 GMT +0000. [18:46:52] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.317 second response time [18:46:52] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.148 second response time [18:47:08] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [18:47:08] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.365 second response time [18:47:52] PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:22] RECOVERY - intp.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:48:58] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:01] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 3.113 second response time [18:49:16] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.29, 1.62, 1.29 [18:50:33] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:42] PROBLEM - mario.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:48] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:50:56] RECOVERY - itora.bunnies1230.com - LetsEncrypt on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Tue 20 Sep 2022 01:50:31 GMT +0000. [18:51:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.81, 2.02, 1.68 [18:51:15] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.058 second response time [18:51:15] RECOVERY - cp20 Stunnel HTTP for puppet141 on cp20 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 2.044 second response time [18:51:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.67, 1.34 [18:51:41] miraheze/CreateWiki - Universal-Omega the build passed. [18:51:45] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:51:45] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Tue 23 Aug 2022 02:37:13 GMT +0000. [18:51:50] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/016672212308 [18:51:51] [02miraheze/MirahezeMagic] 07Universal-Omega 030166722 - Add Agent to T&S preventing log entries being sent to IRC for renaming users [18:51:53] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeMagic [18:51:53] [url] Page not found · GitHub · GitHub | github.com [18:51:54] [02MirahezeMagic] 07Universal-Omega opened pull request 03#356: Add Agent to T&S preventing log entries being sent to IRC for renamin… - 13https://github.com/miraheze/MirahezeMagic/pull/356 [18:51:54] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:51:55] ... [18:52:02] [02MirahezeMagic] 07Universal-Omega edited pull request 03#356: Add Agent to T&S preventing log entries being sent to IRC for renamin… - 13https://github.com/miraheze/MirahezeMagic/pull/356 [18:52:02] ... [18:52:13] [02MirahezeMagic] 07Universal-Omega edited pull request 03#356: Add Agent to T&S preventing log entries being sent to IRC for renaming users - 13https://github.com/miraheze/MirahezeMagic/pull/356 [18:52:13] [url] Page not found · GitHub · GitHub | github.com [18:52:33] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 6% [18:52:56] RECOVERY - allthetropes.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.org' will expire on Tue 23 Aug 2022 01:54:39 GMT +0000. [18:53:02] RECOVERY - wiki.minebox.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minebox.fr' will expire on Fri 16 Sep 2022 18:05:33 GMT +0000. [18:53:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.42, 1.29 [18:53:16] [02MirahezeMagic] 07paladox reviewed pull request 03#356 commit - 13https://github.com/miraheze/MirahezeMagic/pull/356#discussion_r939118183 [18:53:17] [url] Add Agent to T&S preventing log entries being sent to IRC for renaming users by Universal-Omega · Pull Request #356 · miraheze/MirahezeMagic · GitHub | github.com [18:53:32] RECOVERY - grk.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Sat 17 Sep 2022 21:13:49 GMT +0000. [18:53:41] RECOVERY - avas.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avas.wiki' will expire on Mon 10 Oct 2022 04:52:24 GMT +0000. [18:53:55] RECOVERY - www.proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [18:53:59] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.025 second response time [18:54:07] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#356 commit - 13https://github.com/miraheze/MirahezeMagic/pull/356#discussion_r939120503 [18:54:08] [url] Add Agent to T&S preventing log entries being sent to IRC for renaming users by Universal-Omega · Pull Request #356 · miraheze/MirahezeMagic · GitHub | github.com [18:54:40] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [18:55:00] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:55:01] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 39185 bytes in 1.060 second response time [18:55:16] RECOVERY - wiki.sketch.town - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sketch.town' will expire on Thu 13 Oct 2022 04:34:16 GMT +0000. [18:55:45] RECOVERY - www.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikiretet.ga' will expire on Tue 11 Oct 2022 04:42:43 GMT +0000. [18:56:15] RECOVERY - datawiki.online - LetsEncrypt on sslhost is OK: OK - Certificate 'datawiki.online' will expire on Sun 16 Oct 2022 03:33:20 GMT +0000. [18:56:34] RECOVERY - www.puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 20 Oct 2022 16:22:25 GMT +0000. [18:56:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.98, 8.23, 10.03 [18:56:39] RECOVERY - commonwealth.0um.io - LetsEncrypt on sslhost is OK: OK - Certificate 'commonwealth.0um.io' will expire on Thu 20 Oct 2022 18:21:09 GMT +0000. [18:56:42] PROBLEM - cp20 Stunnel HTTP for reports121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:56:58] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [18:57:11] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:57:19] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [18:57:20] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:21] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:21] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.322 second response time [18:57:22] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:57:25] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.062 second response time [18:57:44] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [18:58:02] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:58:13] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:58:24] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [18:58:46] RECOVERY - cp20 Stunnel HTTP for reports121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.054 second response time [18:59:03] RECOVERY - wiki.elgeis.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.elgeis.com' will expire on Fri 02 Sep 2022 12:34:13 GMT +0000. [18:59:08] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Sat 17 Sep 2022 21:13:26 GMT +0000. [18:59:18] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.021 second response time [18:59:37] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.053 second response time [18:59:49] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:00:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:13] PROBLEM - vault.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:13] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.56, 3.35 [19:01:24] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:41] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:15] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:02:46] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:46] PROBLEM - meta.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:57] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sun 02 Oct 2022 10:22:40 GMT +0000. [19:03:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:04] RECOVERY - en.clockupwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.clockupwiki.org' will expire on Sat 10 Sep 2022 21:43:05 GMT +0000. [19:03:15] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.95, 3.28, 3.27 [19:03:24] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 5.084 second response time [19:03:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 21.63, 15.21, 12.29 [19:03:26] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:39] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:03:52] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:04:13] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:04:34] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Tue 23 Aug 2022 00:48:11 GMT +0000. [19:04:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 1.82, 1.92 [19:05:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.295 second response time [19:05:24] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:26] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.344 second response time [19:05:34] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:41] PROBLEM - famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.470 second response time [19:05:51] PROBLEM - smafallfest.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:57] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Wed 24 Aug 2022 03:29:39 GMT +0000. [19:05:57] RECOVERY - wikipanel.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipanel.org' will expire on Sun 02 Oct 2022 04:51:38 GMT +0000. [19:06:07] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:39] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.647 second response time [19:07:04] PROBLEM - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:08] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:07:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.15, 10.70, 11.21 [19:07:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.59, 3.45 [19:07:24] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:08:27] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:36] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Thu 01 Sep 2022 00:46:39 GMT +0000. [19:09:17] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.38, 3.39 [19:09:25] PROBLEM - www.cgradegames.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:39] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.108 second response time [19:09:50] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:57] PROBLEM - orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:04] PROBLEM - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:10:19] PROBLEM - revi.wiki - PositiveSSLDV on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:01] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:11:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 22.26, 14.34, 12.33 [19:11:41] PROBLEM - test3.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:47] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:12:25] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.188 second response time [19:12:39] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:48] PROBLEM - experimyco.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:05] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 5.086 second response time [19:13:17] RECOVERY - wikimas.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimas.kr' will expire on Tue 23 Aug 2022 13:01:24 GMT +0000. [19:13:31] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:36] PROBLEM - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:36] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:45] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:46] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:56] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:05] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:14:21] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:31] PROBLEM - www.agentisai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:38] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.306 second response time [19:14:41] PROBLEM - www.baharna.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.32, 1.63 [19:14:57] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:10] PROBLEM - ffwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:40] PROBLEM - cp21 Stunnel HTTP for reports121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:15:43] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.032 second response time [19:15:48] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [19:15:57] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.020 second response time [19:16:03] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:12] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:25] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.360 second response time [19:16:37] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.183 second response time [19:16:40] PROBLEM - arcanewaterswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:43] PROBLEM - squareenix.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:47] RECOVERY - www.sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [19:17:35] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:48] PROBLEM - wikicompliance.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:10] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.301 second response time [19:18:16] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:18:16] PROBLEM - wiki.recaptime.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:18:17] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/016672212308...72436461e0f9 [19:18:17] [url] Comparing 016672212308...72436461e0f9 · miraheze/MirahezeMagic · GitHub | github.com [19:18:18] [02miraheze/MirahezeMagic] 07Universal-Omega 037243646 - Update MirahezeIRCRCFeedFormatter.php [19:18:20] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#356: Add Agent to T&S preventing log entries being sent to IRC for renaming users - 13https://github.com/miraheze/MirahezeMagic/pull/356 [19:18:20] [url] Page not found · GitHub · GitHub | github.com [19:18:51] [02MirahezeMagic] 07Universal-Omega edited pull request 03#356: Prevent renameuser log entries being sent to IRC for all T&S members - 13https://github.com/miraheze/MirahezeMagic/pull/356 [19:18:52] [url] Page not found · GitHub · GitHub | github.com [19:19:00] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#356 commit - 13https://github.com/miraheze/MirahezeMagic/pull/356#discussion_r939158984 [19:19:01] [url] Prevent renameuser log entries being sent to IRC for all T&S members by Universal-Omega · Pull Request #356 · miraheze/MirahezeMagic · GitHub | github.com [19:19:11] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:51] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:20:12] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:20:32] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:20:34] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 3.073 second response time [19:20:59] PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:05] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Sat 24 Sep 2022 13:22:32 GMT +0000. [19:21:39] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:52] PROBLEM - fortis.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:58] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:22:14] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:22:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.84, 1.57, 1.58 [19:22:43] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:52] RECOVERY - cp21 Stunnel HTTP for reports121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 1.080 second response time [19:22:58] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:09] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:49] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:51] PROBLEM - taswin.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:18] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:23] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:46] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:24:48] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:24:49] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:24:53] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:58] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [19:24:59] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:03] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.93, 3.16, 2.05 [19:25:09] PROBLEM - cp31 Stunnel HTTP for phab121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:25:14] PROBLEM - cp30 Stunnel HTTP for reports121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [19:25:14] PROBLEM - cp31 Stunnel HTTP for puppet141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:25:17] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:25:18] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [19:25:24] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 2.89, 6.79, 11.16 [19:25:25] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [19:25:28] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:50] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.033 second response time [19:26:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:20] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.027 second response time [19:26:29] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 54% [19:26:38] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.043 second response time [19:26:39] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:40] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.54, 1.21, 1.45 [19:26:43] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:26:44] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.336 second response time [19:26:48] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:26:49] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [19:26:51] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [19:26:52] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.196 second response time [19:26:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.243 second response time [19:26:58] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.357 second response time [19:27:07] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:09] RECOVERY - cp31 Stunnel HTTP for phab121 on cp31 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.268 second response time [19:27:10] RECOVERY - roll-of-arms.com - LetsEncrypt on sslhost is OK: OK - Certificate 'roll-of-arms.com' will expire on Thu 27 Oct 2022 15:02:13 GMT +0000. [19:27:12] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:13] RECOVERY - cp30 Stunnel HTTP for reports121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.273 second response time [19:27:14] RECOVERY - cp31 Stunnel HTTP for puppet141 on cp31 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.247 second response time [19:27:15] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.313 second response time [19:27:20] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:27:22] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [19:27:28] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.019 second response time [19:27:29] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:27:31] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:27:33] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:27:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [19:27:37] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:45] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:55] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:00] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:03] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.259 second response time [19:28:28] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:28] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 38% [19:28:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 2.02, 6.03, 10.94 [19:28:50] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.053 second response time [19:28:58] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [19:29:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.21, 1.60, 1.66 [19:29:12] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.065 second response time [19:29:15] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [19:29:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.19, 5.48, 9.70 [19:29:31] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 49% [19:29:36] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.331 second response time [19:29:42] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [19:29:54] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 7.372 second response time [19:30:06] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Mon 22 Aug 2022 22:32:53 GMT +0000. [19:30:12] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.97, 4.77, 3.37 [19:30:25] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sat 17 Sep 2022 20:22:48 GMT +0000. [19:30:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.08, 4.62, 9.82 [19:31:16] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Wed 24 Aug 2022 03:45:41 GMT +0000. [19:31:17] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:19] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:24] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 5.975 second response time [19:31:25] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 11% [19:31:27] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [19:31:35] RECOVERY - wiki.timewisely.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.timewisely.net' will expire on Mon 22 Aug 2022 22:01:18 GMT +0000. [19:31:53] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [19:31:56] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:02] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:11] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.69, 1.45, 3.97 [19:32:51] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:52] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.043 second response time [19:32:58] PROBLEM - cp30 Stunnel HTTP for mw141 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.148 second response time [19:33:00] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.016 second response time [19:33:06] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:33:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [19:33:07] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:11] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:33:14] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [19:33:14] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.288 second response time [19:33:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:15] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.182 second response time [19:33:22] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [19:33:31] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [19:33:49] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.020 second response time [19:33:56] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.013 second response time [19:34:01] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.338 second response time [19:34:05] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.03, 5.28, 3.96 [19:34:05] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.248 second response time [19:34:10] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:12] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:34:13] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:19] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:20] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.031 second response time [19:34:25] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:28] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:31] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 7.198 second response time [19:34:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.20, 2.59, 4.06 [19:34:48] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.283 second response time [19:34:58] RECOVERY - cp30 Stunnel HTTP for mw141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [19:35:00] PROBLEM - wiki.puucraft.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:02] RECOVERY - smafallfest.org - LetsEncrypt on sslhost is OK: OK - Certificate 'smafallfest.org' will expire on Wed 26 Oct 2022 18:44:23 GMT +0000. [19:35:06] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.338 second response time [19:35:12] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.090 second response time [19:35:13] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.354 second response time [19:35:20] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [19:36:02] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.73, 3.90, 3.60 [19:36:21] RECOVERY - www.istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [19:36:45] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.659 second response time [19:36:57] RECOVERY - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chouverse.cludisciples.cyou' will expire on Tue 23 Aug 2022 00:10:19 GMT +0000. [19:36:58] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [19:37:08] PROBLEM - cp21 Stunnel HTTP for mail121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:32] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 2.027 second response time [19:37:45] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:48] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.040 second response time [19:37:53] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:01] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:26] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:28] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Wed 24 Aug 2022 01:38:48 GMT +0000. [19:38:33] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [19:38:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3bba28e58a8f [19:38:45] [02miraheze/mw-config] 07Universal-Omega 033bba28e - phpcs: enable AssignmentInControlStructures rule [19:38:45] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:46] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:47] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [19:38:47] [url] Page not found · GitHub · GitHub | github.com [19:38:48] [02mw-config] 07Universal-Omega opened pull request 03#4851: phpcs: enable AssignmentInControlStructures rule - 13https://github.com/miraheze/mw-config/pull/4851 [19:38:49] [url] Page not found · GitHub · GitHub | github.com [19:38:49] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:09] RECOVERY - cp21 Stunnel HTTP for mail121 on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 1.016 second response time [19:39:10] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Tue 23 Aug 2022 01:42:45 GMT +0000. [19:39:28] RECOVERY - wiki.mostly.vet - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mostly.vet' will expire on Fri 02 Sep 2022 12:38:42 GMT +0000. [19:39:32] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:39:34] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:35] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.434 second response time [19:39:39] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:43] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:47] RECOVERY - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'mcdev.grantlmul.xyz' will expire on Fri 14 Oct 2022 16:18:23 GMT +0000. [19:39:52] [02mw-config] 07Universal-Omega closed pull request 03#4851: phpcs: enable AssignmentInControlStructures rule - 13https://github.com/miraheze/mw-config/pull/4851 [19:39:53] [url] Page not found · GitHub · GitHub | github.com [19:39:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1302e044b497...0aa6c55b5be9 [19:39:54] [url] Comparing 1302e044b497...0aa6c55b5be9 · miraheze/mw-config · GitHub | github.com [19:39:55] [02miraheze/mw-config] 07Universal-Omega 030aa6c55 - phpcs: enable AssignmentInControlStructures rule (#4851) [19:39:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [19:39:58] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [19:39:59] ... [19:40:02] RECOVERY - revi.wiki - PositiveSSLDV on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Thu 29 Dec 2022 23:59:59 GMT +0000. [19:40:04] miraheze/mw-config - Universal-Omega the build passed. [19:40:11] [02miraheze/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/workflow-templates/actions/cache-3.0.6 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/05408e5300f5 [19:40:12] [02miraheze/.github] 07dependabot[bot] 0305408e5 - Bump actions/cache from 3.0.5 to 3.0.6 in /workflow-templates [19:40:14] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/workflow-templates/actions/cache-3.0.6 - 13https://github.com/miraheze/.github [19:40:15] [02.github] 07dependabot[bot] opened pull request 03#14: Bump actions/cache from 3.0.5 to 3.0.6 in /workflow-templates - 13https://github.com/miraheze/.github/pull/14 [19:40:16] ... [19:40:17] [02.github] 07dependabot[bot] labeled pull request 03#14: Bump actions/cache from 3.0.5 to 3.0.6 in /workflow-templates - 13https://github.com/miraheze/.github/pull/14 [19:40:17] [url] Page not found · GitHub · GitHub | github.com [19:40:25] RECOVERY - test3.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:40:36] RECOVERY - www.connorjwat.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.connorjwat.net' will expire on Mon 22 Aug 2022 22:20:46 GMT +0000. [19:40:41] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Fri 14 Oct 2022 16:20:05 GMT +0000. [19:40:56] miraheze/mw-config - Universal-Omega the build passed. [19:41:14] Looking forward to meeting with you 😊 [19:41:35] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 6.019 second response time [19:41:38] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.183 second response time [19:41:43] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.337 second response time [19:41:49] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:49] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:42:09] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.319 second response time [19:42:13] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:18] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [19:42:25] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:32] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:42:35] RECOVERY - wikileague.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wikileague.cf' will expire on Mon 22 Aug 2022 22:21:46 GMT +0000. [19:42:38] PROBLEM - cp21 Stunnel HTTP for matomo131 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:42:45] RECOVERY - mediawiki.ecfin.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'mediawiki.ecfin.eu' will expire on Mon 22 Aug 2022 23:01:19 GMT +0000. [19:42:45] RECOVERY - experimyco.com - LetsEncrypt on sslhost is OK: OK - Certificate 'experimyco.com' will expire on Thu 15 Sep 2022 05:36:04 GMT +0000. [19:42:49] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:43:23] [02mw-config] 07Universal-Omega closed pull request 03#4850: Database: add parsercache cluster to externalLoads - 13https://github.com/miraheze/mw-config/pull/4850 [19:43:24] [url] Page not found · GitHub · GitHub | github.com [19:43:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [19:43:25] RECOVERY - www.baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Sat 17 Sep 2022 21:09:35 GMT +0000. [19:43:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [19:43:27] ... [19:43:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0aa6c55b5be9...b407d303c37e [19:43:28] [url] Comparing 0aa6c55b5be9...b407d303c37e · miraheze/mw-config · GitHub | github.com [19:43:29] [02miraheze/mw-config] 07Universal-Omega 03b407d30 - Database: add parsercache cluster to externalLoads (#4850) [19:43:32] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [19:43:42] RECOVERY - www.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.agentisai.com' will expire on Wed 28 Sep 2022 11:24:06 GMT +0000. [19:43:46] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.98, 5.27, 4.29 [19:43:49] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [19:43:51] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:52] PROBLEM - cp20 Stunnel HTTP for test131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:43:54] [02miraheze/mw-config] 07Universal-Omega pushed 033 commits to 03Universal-Omega-patch-1 [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/605fa95ab050...710501fcdf3a [19:43:54] [url] Comparing 605fa95ab050...710501fcdf3a · miraheze/mw-config · GitHub | github.com [19:43:55] [02miraheze/mw-config] 07Universal-Omega 03710501f - Merge branch 'master' into Universal-Omega-patch-1 [19:43:57] [02mw-config] 07Universal-Omega synchronize pull request 03#4848: MirahezeFunctions: enable phpcs doc comments rules - 13https://github.com/miraheze/mw-config/pull/4848 [19:43:57] [url] Page not found · GitHub · GitHub | github.com [19:44:03] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.072 seconds response time. miraheze.org returns 149.56.140.43,149.56.141.75,2607:5300:201:3100::5ebc,2607:5300:201:3100::929a [19:44:17] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 56% [19:44:24] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:40] miraheze/mw-config - Universal-Omega the build passed. [19:44:43] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [19:44:46] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 24 Aug 2022 16:50:46 GMT +0000. [19:44:53] RECOVERY - ffwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ffwiki.org' will expire on Thu 29 Sep 2022 19:07:40 GMT +0000. [19:44:58] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 17 Sep 2022 21:39:52 GMT +0000. [19:45:05] miraheze/mw-config - Universal-Omega the build passed. [19:45:05] RECOVERY - cp21 Stunnel HTTP for matomo131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 2.212 second response time [19:45:05] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:16] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.018 second response time [19:45:17] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [19:45:23] PROBLEM - cp30 Stunnel HTTP for mw131 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:27] PROBLEM - wikimas.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:35] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [19:45:43] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 27% [19:45:43] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.94, 5.80, 4.62 [19:45:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:50] RECOVERY - cp20 Stunnel HTTP for test131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 1.044 second response time [19:45:56] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.046 second response time [19:45:57] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.324 second response time [19:46:01] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:07] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:46:14] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:46:15] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:19] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [19:46:23] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.006 second response time [19:46:29] RECOVERY - otcg.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'otcg.ml' will expire on Sat 17 Sep 2022 22:12:58 GMT +0000. [19:46:32] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:38] RECOVERY - arcanewaterswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'arcanewaterswiki.org' will expire on Sun 30 Oct 2022 20:54:06 GMT +0000. [19:46:46] PROBLEM - cp20 Stunnel HTTP for phab121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:47] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.027 second response time [19:46:49] RECOVERY - wikicompliance.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicompliance.com' will expire on Fri 07 Oct 2022 11:39:30 GMT +0000. [19:46:53] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:46:54] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [19:47:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.32, 0.34, 0.52 [19:47:25] RECOVERY - wiki.recaptime.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.recaptime.tk' will expire on Wed 26 Oct 2022 23:10:27 GMT +0000. [19:48:13] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [19:48:15] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [19:48:21] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.056 second response time [19:48:26] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 3.048 second response time [19:48:31] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 3.060 second response time [19:48:32] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.316 second response time [19:48:33] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [19:48:35] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [19:48:41] PROBLEM - cp20 Stunnel HTTP for mail121 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:49:05] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [19:49:14] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Sun 02 Oct 2022 11:07:49 GMT +0000. [19:49:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.028 second response time [19:49:34] RECOVERY - cp30 Stunnel HTTP for mw131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.334 second response time [19:49:37] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.97, 4.18, 4.26 [19:49:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.287 second response time [19:49:58] PROBLEM - cp21 Stunnel HTTP for phab121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:14] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is WARNING: WARNING - NGINX Error Rate is 52% [19:50:21] RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Tue 23 Aug 2022 03:06:24 GMT +0000. [19:50:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.75, 1.92, 3.87 [19:50:47] RECOVERY - cp20 Stunnel HTTP for mail121 on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 3.059 second response time [19:50:51] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.052 second response time [19:51:28] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [19:51:43] RECOVERY - fortis.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:51:51] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.049 second response time [19:52:00] RECOVERY - cp21 Stunnel HTTP for phab121 on cp21 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.047 second response time [19:52:03] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:23] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:28] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:30] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.11, 1.30, 3.40 [19:52:31] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:34] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 56% [19:52:40] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:43] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Wed 24 Aug 2022 04:44:37 GMT +0000. [19:52:46] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [19:52:50] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [19:53:10] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:35] PROBLEM - nintendo.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:37] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.016 second response time [19:53:56] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.315 second response time [19:54:12] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 7% [19:54:13] PROBLEM - cp20 conntrack_table_size on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:54:20] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [19:54:21] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [19:54:25] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [19:54:26] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.329 second response time [19:54:29] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.405 second response time [19:54:38] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [19:54:41] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 14% [19:55:23] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.059 second response time [19:55:24] PROBLEM - cp21 Stunnel HTTP for test131 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:27] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [19:55:35] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [19:56:03] PROBLEM - perpustaxaan.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:27] PROBLEM - dragonquest2.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.32, 3.75 [19:56:29] !log [@test131] starting deploy of {'config': True} to all [19:56:30] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:56:36] PROBLEM - solaswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:16] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:20] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:57:24] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:30] PROBLEM - miraheze.ga - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:35] PROBLEM - wiki.minebox.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:38] RECOVERY - cp21 Stunnel HTTP for test131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15940 bytes in 3.048 second response time [19:57:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:45] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:57] PROBLEM - wiki.minecraftathome.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:58:07] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:26] PROBLEM - en.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:42] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [19:58:44] PROBLEM - commonwealth.0um.io - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:58] RECOVERY - vault.agentisai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vault.agentisai.com' will expire on Thu 29 Sep 2022 08:34:24 GMT +0000. [19:59:12] RECOVERY - cp20 conntrack_table_size on cp20 is OK: OK: nf_conntrack is 4 % full [19:59:15] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.375 second response time [19:59:32] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:59:42] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:56] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:12] PROBLEM - christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:34] RECOVERY - meta.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:12:19 GMT +0000. [20:00:37] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Wed 24 Aug 2022 02:47:08 GMT +0000. [20:01:09] RECOVERY - cp20 Stunnel HTTP for phab121 on cp20 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 3.075 second response time [20:01:21] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [20:01:31] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:56] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:02:10] RECOVERY - www.zetapedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zetapedia.com' will expire on Tue 01 Nov 2022 00:05:29 GMT +0000. [20:02:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.92, 10.13, 6.81 [20:02:40] PROBLEM - en.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:01] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Wed 24 Aug 2022 05:10:07 GMT +0000. [20:03:20] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Thu 13 Oct 2022 04:22:47 GMT +0000. [20:03:30] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Tue 23 Aug 2022 01:40:20 GMT +0000. [20:04:08] RECOVERY - gametheorywiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'gametheorywiki.org' will expire on Sun 25 Sep 2022 20:40:02 GMT +0000. [20:04:09] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:21] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Mon 22 Aug 2022 21:48:31 GMT +0000. [20:04:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:44] RECOVERY - wiki.puucraft.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.puucraft.net' will expire on Wed 26 Oct 2022 16:57:15 GMT +0000. [20:05:18] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:42] PROBLEM - txt-dex.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 26.22, 16.20, 9.64 [20:06:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.486 second response time [20:06:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 10.46, 7.83 [20:06:59] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 13 day(s) (Fri 19 Aug 2022 17:20:05 GMT +0000). [20:07:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.051 second response time [20:07:42] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:53] RECOVERY - www.cgradegames.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.cgradegames.net' will expire on Wed 02 Nov 2022 19:26:30 GMT +0000. [20:08:00] PROBLEM - wiki.virtualjet.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.43, 9.27, 7.72 [20:08:48] PROBLEM - www.istpcomputing.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:52] !log [@mwtask141] starting deploy of {'config': True} to all [20:09:54] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw121.miraheze.org [20:09:57] PROBLEM - orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:11:48] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:12:03] RECOVERY - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.phgalaxy.eu.org' will expire on Mon 10 Oct 2022 06:42:26 GMT +0000. [20:12:09] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [20:12:53] PROBLEM - wiki.newnormal.fans - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:33] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:13:56] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 2.046 second response time [20:14:57] PROBLEM - source.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:12] RECOVERY - wikimas.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimas.kr' will expire on Tue 23 Aug 2022 13:01:24 GMT +0000. [20:15:27] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3405 bytes in 0.019 second response time [20:16:12] PROBLEM - civwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:55] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 24 Aug 2022 05:02:45 GMT +0000. [20:18:00] PROBLEM - cp20 Stunnel HTTP for matomo131 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:18:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.17, 11.64, 9.04 [20:18:41] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:13] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:07] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:12] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 9.97, 8.73 [20:21:10] PROBLEM - electowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:29] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:12] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:22:15] PROBLEM - cp21 PowerDNS Recursor on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:23:12] RECOVERY - hellointernet.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [20:23:43] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:13] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [20:24:18] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.016 second response time [20:24:18] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:24:21] RECOVERY - cp21 PowerDNS Recursor on cp21 is OK: DNS OK: 0.175 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,51.195.220.68 [20:25:49] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Sun 02 Oct 2022 10:05:42 GMT +0000. [20:26:28] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [20:26:30] RECOVERY - wiki.minecraftathome.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minecraftathome.com' will expire on Wed 24 Aug 2022 02:32:00 GMT +0000. [20:26:39] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Wed 24 Aug 2022 04:25:02 GMT +0000. [20:26:45] RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [20:26:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:27:16] RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Fri 02 Sep 2022 12:38:19 GMT +0000. [20:27:21] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Mon 26 Sep 2022 13:36:29 GMT +0000. [20:27:37] RECOVERY - wiki.minebox.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minebox.fr' will expire on Fri 16 Sep 2022 18:05:33 GMT +0000. [20:27:51] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:04:52 GMT +0000. [20:28:00] RECOVERY - commonwealth.0um.io - LetsEncrypt on sslhost is OK: OK - Certificate 'commonwealth.0um.io' will expire on Thu 20 Oct 2022 18:21:09 GMT +0000. [20:28:10] RECOVERY - cp20 Stunnel HTTP for matomo131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.222 second response time [20:28:21] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 13.05, 9.89, 7.33 [20:28:29] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Mon 22 Aug 2022 23:35:07 GMT +0000. [20:28:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:29:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.36, 4.49, 3.68 [20:30:21] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.02, 9.47, 7.50 [20:31:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.96, 3.59 [20:31:53] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:55] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Tue 23 Aug 2022 01:21:14 GMT +0000. [20:33:58] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [20:34:23] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:34:30] RECOVERY - txt-dex.com - LetsEncrypt on sslhost is OK: OK - Certificate 'txt-dex.com' will expire on Sat 29 Oct 2022 14:13:35 GMT +0000. [20:34:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.81, 14.57, 10.61 [20:35:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.66, 2.92, 3.26 [20:37:11] RECOVERY - wiki.virtualjet.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Sun 25 Sep 2022 21:20:59 GMT +0000. [20:37:35] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [20:37:38] RECOVERY - www.istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Tue 11 Oct 2022 04:37:06 GMT +0000. [20:38:01] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Wed 24 Aug 2022 03:18:35 GMT +0000. [20:38:09] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [20:38:31] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 26 Oct 2022 18:24:04 GMT +0000. [20:39:10] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [20:40:08] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:40:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.67, 11.94, 10.79 [20:42:40] RECOVERY - wiki.newnormal.fans - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.newnormal.fans' will expire on Fri 14 Oct 2022 16:20:05 GMT +0000. [20:43:11] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw121 mw131 [20:43:46] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:58] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [20:43:58] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [20:44:07] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:07] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [20:44:07] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.478 second response time [20:44:13] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:21] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:33] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.35, 9.21, 9.90 [20:44:55] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:57] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.477 second response time [20:44:58] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 17 Sep 2022 21:39:52 GMT +0000. [20:46:03] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.694 second response time [20:46:06] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.821 second response time [20:46:10] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.082 second response time [20:46:10] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.457 second response time [20:46:13] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.412 second response time [20:46:27] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.622 second response time [20:46:32] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [20:47:10] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.046 second response time [20:47:35] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw132 mw142 [20:47:53] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:31] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'betaheze.org' expires in 13 day(s) (Fri 19 Aug 2022 17:58:05 GMT +0000). [20:49:08] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.201 second response time [20:49:10] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Sat 01 Oct 2022 15:33:56 GMT +0000. [20:49:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.76, 9.72, 11.77 [20:49:29] PROBLEM - cloudwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:35] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [20:50:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:50:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 7.145 second response time [20:50:45] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:06] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [20:51:08] RECOVERY - nintendo.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [20:53:12] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:39] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [20:54:11] RECOVERY - solaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'solaswiki.org' will expire on Sat 03 Sep 2022 15:10:56 GMT +0000. [20:54:15] PROBLEM - www.trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:54:30] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [20:54:57] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:54:59] RECOVERY - perpustaxaan.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'perpustaxaan.wiki' will expire on Tue 11 Oct 2022 04:34:44 GMT +0000. [20:55:37] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [20:56:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 3.99, 3.30 [20:59:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 20.14, 13.60, 12.24 [20:59:37] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Wed 24 Aug 2022 04:17:42 GMT +0000. [21:00:07] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:02:04] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Wed 28 Sep 2022 05:13:53 GMT +0000. [21:04:18] PROBLEM - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:18] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:19] PROBLEM - cp20 Stunnel HTTP for mwtask141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:45] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw142 [21:08:21] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:00] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:09:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.53, 12.83, 10.68 [21:09:38] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [21:09:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:06] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:52] RECOVERY - cp20 Stunnel HTTP for mwtask141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 3.040 second response time [21:12:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.27, 5.22, 3.06 [21:12:32] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:12:37] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:13:02] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 204s [21:13:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.85, 3.97 [21:13:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:57] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:12] PROBLEM - squareenixwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:30] PROBLEM - sharqedia.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:46] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.058 second response time [21:17:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.05, 10.52, 10.73 [21:17:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.99, 9.12, 11.82 [21:17:49] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sun 30 Oct 2022 08:17:32 GMT +0000. [21:18:11] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:19] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 3.063 second response time [21:18:27] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.35, 7.13, 4.60 [21:18:33] RECOVERY - cloudwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cloudwiki.org' will expire on Sun 23 Oct 2022 16:49:57 GMT +0000. [21:19:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.10, 9.11, 10.18 [21:19:30] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Tue 23 Aug 2022 00:50:24 GMT +0000. [21:20:09] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:20:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.91, 7.22, 4.94 [21:21:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 2.65, 3.33 [21:22:21] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [21:22:22] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:22:27] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.62, 6.62, 5.01 [21:23:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.46, 7.22, 10.05 [21:23:28] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [21:25:29] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [21:27:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 9.45, 10.33 [21:28:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.46, 11.31, 10.45 [21:29:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 4.14, 3.70 [21:29:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.30, 11.60, 11.07 [21:29:37] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:30:03] [02miraheze/MirahezeMagic] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/cache-3.0.6 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/252a512d5c9b [21:30:05] [02miraheze/MirahezeMagic] 07dependabot[bot] 03252a512 - Bump actions/cache from 3.0.5 to 3.0.6 [21:30:06] [02MirahezeMagic] 07dependabot[bot] created branch 03dependabot/github_actions/actions/cache-3.0.6 - 13https://github.com/miraheze/MirahezeMagic [21:30:07] [url] Page not found · GitHub · GitHub | github.com [21:30:08] [02MirahezeMagic] 07dependabot[bot] opened pull request 03#357: Bump actions/cache from 3.0.5 to 3.0.6 - 13https://github.com/miraheze/MirahezeMagic/pull/357 [21:30:08] [url] Page not found · GitHub · GitHub | github.com [21:30:09] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#357: Bump actions/cache from 3.0.5 to 3.0.6 - 13https://github.com/miraheze/MirahezeMagic/pull/357 [21:30:10] ... [21:30:11] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#357: Bump actions/cache from 3.0.5 to 3.0.6 - 13https://github.com/miraheze/MirahezeMagic/pull/357 [21:30:11] [url] Page not found · GitHub · GitHub | github.com [21:30:25] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:31:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.21, 11.87, 11.27 [21:32:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.78, 11.69, 10.81 [21:33:13] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Tue 23 Aug 2022 08:56:40 GMT +0000. [21:33:13] RECOVERY - www.marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Wed 24 Aug 2022 01:59:18 GMT +0000. [21:33:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.01, 13.50, 11.92 [21:35:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.73, 3.74 [21:35:56] more 503 errors than usual -- does anyone know why this is [21:38:12] RECOVERY - feathercare.net - LetsEncrypt on sslhost is OK: OK - Certificate 'feathercare.net' will expire on Sat 27 Aug 2022 14:00:44 GMT +0000. [21:38:56] miraheze/MirahezeMagic - dependabot[bot] the build passed. [21:40:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.11, 12.23, 11.10 [21:41:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 2.70, 3.28 [21:42:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.58, 11.97, 11.16 [21:43:58] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Mon 10 Oct 2022 04:58:16 GMT +0000. [21:44:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:45:01] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Sat 01 Oct 2022 04:20:30 GMT +0000. [21:47:11] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Tue 23 Aug 2022 07:40:05 GMT +0000. [21:50:22] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [21:52:01] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Sat 24 Sep 2022 13:22:32 GMT +0000. [21:52:42] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Tue 23 Aug 2022 08:34:15 GMT +0000. [21:53:00] RECOVERY - www.trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Mon 10 Oct 2022 06:29:05 GMT +0000. [21:57:31] RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 17 Sep 2022 18:34:21 GMT +0000. [21:58:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.14, 9.52, 10.03 [21:59:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.56, 3.24 [21:59:44] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:59:44] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:59:46] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [21:59:46] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:50] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [21:59:51] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [21:59:51] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:55] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:56] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:59:56] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:58] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:00:02] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:05] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:46] PROBLEM - cp30 Stunnel HTTP for mw142 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:46] PROBLEM - cp20 Stunnel HTTP for mw142 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:50] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:51] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:59] PROBLEM - cp31 Stunnel HTTP for test131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:00:59] PROBLEM - cp31 Stunnel HTTP for matomo131 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:01:03] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:07] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:14] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:25] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:01:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.51, 2.52, 2.89 [22:01:40] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [22:01:42] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.021 second response time [22:01:44] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.227 second response time [22:01:45] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 1.542 second response time [22:01:51] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [22:01:53] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.470 second response time [22:01:55] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [22:01:57] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.203 second response time [22:02:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.55, 1.88, 3.97 [22:02:45] RECOVERY - cp20 Stunnel HTTP for mw142 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [22:02:46] RECOVERY - cp30 Stunnel HTTP for mw142 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [22:02:48] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [22:02:50] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.369 second response time [22:02:58] RECOVERY - cp31 Stunnel HTTP for test131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.314 second response time [22:02:59] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 0.729 second response time [22:03:01] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [22:03:01] https://twitter.com/miraheze/status/1555643574284320768 ah [22:03:02] [Twitter] Miraheze (@miraheze): We are aware of recent issues when trying to access wikis, including slowness and 503s. This is due to ongoing server file storage server migration which causes server strain. We apologize for the inconvenience. Please check https://status.miraheze.wiki for more updates. | 0 RTs | 6 ♥s | Posted: 2022-08-05 - 20:55:05BST [22:03:02] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 4.499 second response time [22:04:29] PROBLEM - cp20 Stunnel HTTP for mw132 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:04:29] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:36] PROBLEM - cp31 Stunnel HTTP for mw132 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw121 mw131 mw132 [22:04:54] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [22:05:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 3.65, 7.63, 10.88 [22:05:30] PROBLEM - gluster102 SSH on gluster102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:49] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:05:56] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.336 second response time [22:06:00] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:14] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.336 second response time [22:06:16] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:17] must be a lot of files transferring [22:06:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.07, 0.95, 3.11 [22:06:18] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:06:20] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:21] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 46% [22:06:23] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:26] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:28] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:06:35] RECOVERY - cp31 Stunnel HTTP for mw132 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.701 second response time [22:06:40] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 10.14, 7.34, 4.10 [22:06:42] PROBLEM - cloud10 Current Load on cloud10 is WARNING: WARNING - load average: 46.88, 35.96, 22.95 [22:06:46] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [22:06:48] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is WARNING: WARNING - NGINX Error Rate is 54% [22:06:51] PROBLEM - cp20 Stunnel HTTP for mw141 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:58] PROBLEM - gluster102 PowerDNS Recursor on gluster102 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:07:19] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [22:07:24] RECOVERY - gluster102 SSH on gluster102 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [22:07:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 1.84, 5.64, 9.76 [22:07:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [22:07:45] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.998 second response time [22:07:47] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [22:07:54] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.006 second response time [22:08:03] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.022 second response time [22:08:11] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.390 second response time [22:08:15] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [22:08:21] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [22:08:22] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [22:08:23] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [22:08:24] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.029 second response time [22:08:25] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [22:08:27] RECOVERY - cp20 Stunnel HTTP for mw132 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [22:08:35] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.009 second response time [22:08:37] RECOVERY - cloud10 Current Load on cloud10 is OK: OK - load average: 24.43, 32.19, 23.17 [22:08:48] RECOVERY - cp20 Stunnel HTTP for mw141 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.025 second response time [22:08:48] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 4% [22:08:58] RECOVERY - gluster102 PowerDNS Recursor on gluster102 is OK: DNS OK: 0.428 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [22:09:01] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [22:10:06] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [22:10:07] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.504 second response time [22:10:21] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 4% [22:10:28] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.433 second response time [22:10:31] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.568 second response time [22:10:34] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.36, 4.41, 3.67 [22:11:00] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 3% [22:11:40] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [22:12:34] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [22:12:52] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [22:13:44] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:13:59] PROBLEM - cp21 Stunnel HTTP for mw141 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:05] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:08] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:16] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:18] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [22:14:23] PROBLEM - cp21 Stunnel HTTP for mw142 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:24] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:26] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:28] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:28] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:33] PROBLEM - cp21 Stunnel HTTP for mw131 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:34] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:36] PROBLEM - cp21 Stunnel HTTP for mw132 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [22:14:41] PROBLEM - cp31 Stunnel HTTP for mw131 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:47] RECOVERY - sharqedia.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'sharqedia.ml' will expire on Mon 03 Oct 2022 13:40:45 GMT +0000. [22:15:46] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:46] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:50] PROBLEM - cp31 Stunnel HTTP for mw142 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:59] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [22:16:06] RECOVERY - cp21 Stunnel HTTP for mw141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.562 second response time [22:16:07] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.331 second response time [22:16:13] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [22:16:20] RECOVERY - cp21 Stunnel HTTP for mw142 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [22:16:21] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.302 second response time [22:16:21] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.67, 5.28, 4.35 [22:16:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.356 second response time [22:16:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29512 bytes in 0.284 second response time [22:16:27] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.329 second response time [22:16:30] RECOVERY - cp21 Stunnel HTTP for mw131 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [22:16:32] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.026 second response time [22:16:34] RECOVERY - cp21 Stunnel HTTP for mw132 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.016 second response time [22:16:41] RECOVERY - cp31 Stunnel HTTP for mw131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.353 second response time [22:16:51] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw142 [22:17:40] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 4 backends are down. mw121 mw122 mw132 mw141 [22:17:41] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.019 second response time [22:17:45] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.401 second response time [22:17:46] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [22:17:49] RECOVERY - cp31 Stunnel HTTP for mw142 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.305 second response time [22:18:18] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.42, 3.85, 3.94 [22:18:24] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [22:19:41] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [22:20:20] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [22:20:50] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [22:24:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.55, 6.22, 4.12 [22:26:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.53, 12.53, 10.38 [22:41:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.36, 2.76 [22:43:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.35, 3.35, 2.83 [22:47:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.49, 2.98 [22:49:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.95, 2.84 [22:53:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.41, 10.96, 11.81 [22:57:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.16, 11.47, 11.71 [22:59:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.84, 3.20 [22:59:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 11.80, 11.87 [23:01:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.67, 3.03, 2.98 [23:03:21] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.80, 12.78, 12.19 [23:04:48] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw121 [23:05:20] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [23:05:55] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [23:06:48] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [23:07:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.83, 10.25, 11.43 [23:07:55] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [23:11:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.38, 10.63, 11.24 [23:13:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 10.01, 10.93 [23:19:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.62, 12.73, 11.76 [23:22:11] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.83, 1.39 [23:23:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.53, 10.03, 8.84 [23:24:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.63, 1.36 [23:25:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.18, 9.05, 8.61 [23:27:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.07, 11.76, 11.69 [23:29:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.88, 13.70, 12.42 [23:30:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 4.14, 3.11 [23:30:37] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4223 MB (10% inode=96%); [23:34:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.79, 3.21 [23:39:52] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.48, 5.49, 3.25 [23:41:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 2.93, 3.06 [23:43:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.01, 11.61, 9.69 [23:43:52] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.86, 5.99, 4.00 [23:45:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.93, 9.78, 9.26