[00:01:18] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.95, 11.16, 9.23 [00:03:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.82, 1.45 [00:03:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.53, 3.07 [00:04:45] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 7.64, 12.52, 10.32 [00:05:18] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 27.38, 16.22, 11.38 [00:05:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.79, 1.42, 1.34 [00:06:43] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.14, 10.11, 9.70 [00:07:12] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 33.26, 17.11, 10.39 [00:07:30] PROBLEM - cp31 Stunnel HTTP for mw141 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:44] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.83, 10.70, 8.19 [00:07:50] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 25035 bytes in 7.870 second response time [00:09:18] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 6.77, 10.56, 10.14 [00:09:30] RECOVERY - cp31 Stunnel HTTP for mw141 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.249 second response time [00:09:30] PROBLEM - mw141 nutcracker process on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 115 (nutcracker), command name 'nutcracker' [00:09:43] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.45, 8.25, 7.58 [00:09:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.14, 2.89, 2.93 [00:11:07] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw141 [00:11:10] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 3.91, 11.40, 9.75 [00:11:18] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.85, 9.80, 9.89 [00:13:07] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [00:13:10] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.49, 9.67, 9.32 [00:25:40] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.03, 5.21, 4.02 [00:29:40] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.60, 4.82, 4.18 [00:39:40] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.48, 4.91, 4.33 [00:41:40] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 4.87, 4.88, 4.39 [00:47:40] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.19, 5.10, 4.64 [00:48:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.18, 2.63 [00:49:40] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.92, 4.75, 4.58 [00:50:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 2.85, 2.58 [00:55:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3d2e7e0699c4 [00:55:32] [02miraheze/mw-config] 07Universal-Omega 033d2e7e0 - Add `$wgUseRCPatrol` to ManageWikiSettings [00:55:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [00:55:34] [url] Page not found · GitHub · GitHub | github.com [00:55:37] [02mw-config] 07Universal-Omega opened pull request 03#4886: T9716: add `$wgUseRCPatrol` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4886 [00:55:38] [url] Page not found · GitHub · GitHub | github.com [00:56:48] miraheze/mw-config - Universal-Omega the build passed. [00:57:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3d2e7e0699c4...593c7c0fd9cd [00:57:01] [url] Comparing 3d2e7e0699c4...593c7c0fd9cd · miraheze/mw-config · GitHub | github.com [00:57:02] [02miraheze/mw-config] 07Universal-Omega 03593c7c0 - Update LocalSettings.php [00:57:04] [02mw-config] 07Universal-Omega synchronize pull request 03#4886: T9716: add `$wgUseRCPatrol` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4886 [00:57:04] [url] Page not found · GitHub · GitHub | github.com [00:58:02] miraheze/mw-config - Universal-Omega the build passed. [00:59:40] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.14, 4.95, 4.66 [01:01:40] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 4.52, 4.81, 4.64 [01:02:03] !log reboot mw141 (nutcracker would not start) [01:02:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:21] [02mw-config] 07Universal-Omega closed pull request 03#4886: T9716: add `$wgUseRCPatrol` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4886 [01:02:21] [url] Page not found · GitHub · GitHub | github.com [01:02:22] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [01:02:23] ... [01:02:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/4a3d730ed5a2...f467ba6dfdce [01:02:24] [url] Comparing 4a3d730ed5a2...f467ba6dfdce · miraheze/mw-config · GitHub | github.com [01:02:25] [02miraheze/mw-config] 07Universal-Omega 03f467ba6 - T9716: add `$wgUseRCPatrol` to ManageWikiSettings (#4886) [01:02:27] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:03:17] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [01:03:23] RECOVERY - mw141 nutcracker process on mw141 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [01:03:29] miraheze/mw-config - Universal-Omega the build passed. [01:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:39] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [01:03:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:04:03] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 29514 bytes in 2.470 second response time [01:05:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.46, 3.14, 2.22 [01:07:22] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.41, 2.42, 2.07 [01:10:17] [02mw-config] 07Universal-Omega commented on pull request 03#4884: T9716: add wgUseRCPatrol to ManageWiki - 13https://github.com/miraheze/mw-config/pull/4884#issuecomment-1236227159 [01:10:18] [url] T9716: add wgUseRCPatrol to ManageWiki by ugochimobi · Pull Request #4884 · miraheze/mw-config · GitHub | github.com [01:10:18] [02mw-config] 07Universal-Omega closed pull request 03#4884: T9716: add wgUseRCPatrol to ManageWiki - 13https://github.com/miraheze/mw-config/pull/4884 [01:10:19] [url] Page not found · GitHub · GitHub | github.com [01:15:26] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405765056610107 seconds [01:27:43] !log [@test131] starting deploy of {'config': True} to all [01:27:44] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [01:27:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.87, 1.60, 1.31 [01:30:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.48, 1.50, 1.30 [01:34:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.17, 2.42, 1.61 [01:38:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.87, 1.58 [01:40:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.70, 1.56 [01:45:16] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:52:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 1.88, 1.60 [01:53:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.63, 3.00 [01:54:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.79, 1.60 [01:55:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 3.13, 2.89 [01:56:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.50, 1.51 [01:56:33] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.45, 1.86, 1.54 [01:58:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.84, 1.91, 1.61 [02:00:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.32, 1.69, 1.56 [02:12:41] PROBLEM - itora.bunnies1230.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itora.bunnies1230.com' expires in 15 day(s) (Tue 20 Sep 2022 01:50:31 GMT +0000). [02:19:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/26c545d0f222...1c5105770151 [02:19:31] [url] Comparing 26c545d0f222...1c5105770151 · miraheze/ssl · GitHub | github.com [02:19:32] [02miraheze/ssl] 07MirahezeSSLBot 031c51057 - Bot: Update SSL cert for itora.bunnies1230.com [02:24:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.45, 3.98, 2.49 [02:25:30] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.94, 11.65, 8.96 [02:27:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.17, 5.21, 3.75 [02:27:24] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.15, 11.47, 9.24 [02:27:34] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.12, 2.20, 1.63 [02:28:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:06] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:19] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.66, 9.40, 8.75 [02:29:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.78, 1.54 [02:30:53] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29515 bytes in 9.026 second response time [02:31:12] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.630 second response time [02:31:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.41, 1.43 [02:31:41] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.698 second response time [02:32:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.80, 3.58 [02:35:16] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:39] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:01] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 4.40, 3.85 [02:43:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29514 bytes in 8.232 second response time [02:47:40] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.70, 4.17, 3.34 [02:47:51] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:29] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 9.675 second response time [02:50:32] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 9.435 second response time [02:56:11] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29515 bytes in 6.886 second response time [02:56:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.57, 1.60, 1.97 [03:00:06] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:16] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.60, 2.05, 2.00 [03:00:34] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:10] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 9.733 second response time [03:02:22] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.187 second response time [03:02:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29514 bytes in 8.979 second response time [03:03:24] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.37, 3.93, 3.94 [03:04:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.93, 1.97 [03:05:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.44, 3.95, 3.93 [03:06:30] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:49] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:59] PROBLEM - savagepedia.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - savagepedia.wiki reverse DNS resolves to any-in-2001-4860-4802-34--15.1e100.net [03:07:03] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.55, 3.61, 3.80 [03:10:41] RECOVERY - itora.bunnies1230.com - LetsEncrypt on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Sat 03 Dec 2022 01:19:24 GMT +0000. [03:11:19] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.17, 11.41, 8.85 [03:13:18] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.65, 9.63, 8.51 [03:13:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.13, 4.42, 4.04 [03:14:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.41, 1.66 [03:15:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.99, 3.78, 3.85 [03:16:49] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 6.571 second response time [03:17:10] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 3.268 second response time [03:17:26] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29515 bytes in 4.921 second response time [03:21:22] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.02, 2.56, 3.27 [03:21:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29515 bytes in 2.189 second response time [03:23:48] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 201 bytes in 0.242 second response time [03:24:22] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:23] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 4.909 second response time [03:26:58] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 29515 bytes in 1.451 second response time [03:27:48] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.232 second response time [03:27:50] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.18, 1.78, 1.50 [03:29:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.09, 3.45, 3.41 [03:29:47] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.62, 1.68, 1.50 [03:31:22] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.15, 3.15, 3.32 [03:39:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.08, 3.75, 3.44 [03:41:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.32, 3.77, 3.50 [03:42:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.00, 1.76, 1.59 [03:43:22] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.38, 3.25, 3.34 [03:44:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.44, 1.63, 1.56 [03:47:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.36, 3.97 [03:49:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.55, 1.83, 1.50 [03:51:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.05, 1.49, 1.41 [03:52:14] PROBLEM - savagepedia.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [03:57:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.76, 3.83 [03:59:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.78, 3.83 [04:13:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.54, 2.67, 3.26 [04:16:01] PROBLEM - en.clockupwiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.clockupwiki.org All nameservers failed to answer the query. [04:39:27] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.77, 1.38, 0.91 [04:41:27] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.13, 1.32, 0.95 [04:43:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.76, 3.34 [04:45:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.40, 3.25 [04:52:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.70, 1.41 [04:54:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.26, 1.54, 1.39 [04:57:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.93, 3.12, 2.73 [04:57:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.17, 3.46 [04:59:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.74, 3.40 [05:01:22] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 3.08, 3.18, 2.88 [05:03:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 4.03, 3.57 [05:05:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.51, 3.43 [05:17:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.75, 3.65 [05:19:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.91, 3.71 [05:20:44] RECOVERY - savagepedia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'savagepedia.wiki' will expire on Sun 06 Nov 2022 04:40:51 GMT +0000. [05:21:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.76, 3.68 [05:22:55] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.09, 1.66, 1.46 [05:23:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.85, 3.71 [05:24:53] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.57, 1.62, 1.47 [05:28:44] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.12, 2.01, 1.66 [05:29:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.92, 3.78 [05:30:42] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.56, 1.82, 1.63 [05:32:39] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.33, 1.67, 1.60 [05:34:59] RECOVERY - savagepedia.wiki - reverse DNS on sslhost is OK: SSL OK - savagepedia.wiki reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK [05:41:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 2.93, 3.35 [05:53:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.21, 3.09 [05:55:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 2.87, 2.98 [05:59:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.73, 3.35 [06:01:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.94, 3.46 [06:02:54] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 17.55, 13.10, 8.54 [06:03:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.11, 1.85, 1.46 [06:03:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.88, 3.50 [06:03:45] PROBLEM - cp30 Stunnel HTTP for mw132 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.34, 1.70, 1.45 [06:05:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.40, 3.37 [06:05:40] RECOVERY - cp30 Stunnel HTTP for mw132 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.235 second response time [06:08:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.91, 1.51, 1.30 [06:08:51] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 15.85, 9.20, 5.03 [06:08:53] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 4.39, 9.88, 9.09 [06:09:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:14] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:34] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.07, 1.66, 1.38 [06:10:51] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.91, 7.34, 4.86 [06:11:13] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29514 bytes in 6.368 second response time [06:11:14] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 5.777 second response time [06:12:34] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.96, 1.77, 1.46 [06:12:51] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.94, 5.70, 4.55 [06:14:33] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.42, 2.00, 1.58 [06:15:39] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.50, 3.74, 2.61 [06:18:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.55, 3.54, 3.23 [06:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.49, 3.25 [06:22:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.66, 2.88, 3.06 [06:23:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.45, 1.46, 1.92 [06:27:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.00, 1.61 [06:31:16] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:31] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:19] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.493 second response time [06:33:37] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.562 second response time [06:37:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.59, 3.26 [06:39:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.56, 4.35, 3.58 [06:40:38] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.85, 3.52, 3.06 [06:41:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.74, 3.45 [06:42:36] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.73, 2.91, 2.89 [06:43:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.92, 3.27, 3.33 [06:57:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.51, 3.25 [06:59:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.74, 3.36 [07:01:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.53, 3.34 [07:03:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.30, 3.27 [07:07:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.54, 3.38 [07:09:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.60, 2.83, 3.14 [07:15:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 4.45, 3.71 [07:17:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.83, 3.40, 3.41 [07:19:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 2.84, 3.20 [07:30:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.12, 1.44, 1.90 [07:34:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.08, 1.22, 1.70 [08:11:18] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:02] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:13:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 29514 bytes in 6.519 second response time [08:15:00] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 1.068 second response time [08:50:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.38, 2.81, 3.83 [08:52:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.75, 1.59, 1.40 [08:54:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.66, 1.60, 1.43 [08:54:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.37, 3.81 [08:56:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.00, 3.61 [08:58:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.50, 1.76, 1.55 [08:58:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.22, 3.59 [09:00:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.30, 1.60, 1.52 [09:00:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.22, 3.56 [09:02:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.87, 5.54, 4.34 [09:02:52] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406592130661011 seconds [09:03:52] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:05:49] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 0.062 second response time [09:32:03] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp31.miraheze.org - CNAME FLAT [10:25:27] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:27:27] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 3.076 second response time [10:31:20] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.80, 1.73, 1.52 [10:41:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:33] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.46, 1.60, 1.50 [10:43:37] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 0.042 second response time [10:44:25] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 3.33, 2.23 [10:48:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:50:36] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 2.053 second response time [10:54:05] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.62, 3.99, 3.19 [10:55:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:01:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:03:44] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.45, 3.14, 3.16 [11:04:00] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:04:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.23, 3.91 [11:05:58] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 0.041 second response time [11:06:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.48, 4.10, 4.13 [11:08:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.53, 3.90 [11:12:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.96, 4.63, 4.18 [11:18:05] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fa805880f8b0...5b04ea275b3e [11:18:05] [url] Comparing fa805880f8b0...5b04ea275b3e · miraheze/puppet · GitHub | github.com [11:18:06] [02miraheze/puppet] 07JohnFLewis 035b04ea2 - prometheus: temporarily remove cp* blackbox monitoring [11:18:29] Reception123: CosmicAlpha ^^ massive change to reduce mw122's footprint [11:18:53] removes around 64k requests a day [11:19:47] JohnLewis: wow [11:19:48] That change has been around since 2021 when SPF introduced it, but it's likely causing more damage than good right now [11:20:14] so in this case the monitoring is making things worse? [11:21:10] Yeah, in this case, we are monitoring too intensively and gaining nothing out of it. We're monitoring for the sake of data at the cost of stability which is why I've removed it. It was paladox actually that identified it :) [11:21:22] but he didn't want to remove it without asking me as it is monitoring [11:25:54] Let's see how things go then :) [11:30:31] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 21 Sep 2022 09:01:23 GMT +0000. [11:32:12] JohnLewis: that's going to be a huge saving [11:32:34] CA was wondering why mw12* were doing so much worse [11:32:58] I was wondering the same tbh [11:33:15] That should now hopefully see an improvement [11:38:19] Should be visible fairly quick [11:38:55] JohnLewis: Grafana is broken [11:39:16] What do you mean? [11:39:36] JohnLewis: If you're seeing this Grafana has failed to load its application files [11:39:45] Oh back [11:39:45] just refresh [11:39:56] it's one of many problems on a backburner for me today [11:41:05] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:10] Ok [11:52:45] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:54:18] PROBLEM - commons.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.35, 1.70, 1.22 [11:54:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.24, 3.95, 2.72 [11:54:46] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 324 bytes in 0.047 second response time [11:56:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.34, 1.15 [11:59:23] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.92, 3.81, 2.60 [12:05:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a27e64cd2da2 [12:05:01] [02miraheze/puppet] 07paladox 03a27e64c - varnish::stunnel4: fix check for mon141 [12:05:03] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [12:05:03] [url] Page not found · GitHub · GitHub | github.com [12:05:04] [02puppet] 07paladox opened pull request 03#2802: varnish::stunnel4: fix check for mon141 - 13https://github.com/miraheze/puppet/pull/2802 [12:05:04] [url] Page not found · GitHub · GitHub | github.com [12:05:22] [02puppet] 07paladox closed pull request 03#2802: varnish::stunnel4: fix check for mon141 - 13https://github.com/miraheze/puppet/pull/2802 [12:05:23] [url] Page not found · GitHub · GitHub | github.com [12:05:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5b04ea275b3e...4107a2541b4f [12:05:24] [url] Comparing 5b04ea275b3e...4107a2541b4f · miraheze/puppet · GitHub | github.com [12:05:25] [02miraheze/puppet] 07paladox 034107a25 - varnish::stunnel4: fix check for mon141 (#2802) [12:05:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [12:05:28] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [12:05:29] [url] Page not found · GitHub · GitHub | github.com [12:07:29] PROBLEM - dc.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:29] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [12:07:29] [url] Page not found · GitHub · GitHub | github.com [12:07:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8fb3d92c1b95 [12:07:32] [02miraheze/puppet] 07paladox 038fb3d92 - varnish::stunnel4: fix check for matomo131 [12:07:33] [02puppet] 07paladox opened pull request 03#2803: varnish::stunnel4: fix check for matomo131 - 13https://github.com/miraheze/puppet/pull/2803 [12:07:34] ... [12:07:42] [02puppet] 07paladox closed pull request 03#2803: varnish::stunnel4: fix check for matomo131 - 13https://github.com/miraheze/puppet/pull/2803 [12:07:42] ... [12:07:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [12:07:45] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [12:07:45] ... [12:07:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4107a2541b4f...22b64c7bd8d2 [12:07:48] [02miraheze/puppet] 07paladox 0322b64c7 - varnish::stunnel4: fix check for matomo131 (#2803) [12:07:49] [url] Comparing 4107a2541b4f...22b64c7bd8d2 · miraheze/puppet · GitHub | github.com [12:08:02] PROBLEM - test1.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:16] PROBLEM - notes.mridulpm.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'notes.mridulpm.in' expires in 15 day(s) (Tue 20 Sep 2022 12:00:20 GMT +0000). [12:09:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.23, 3.97, 3.39 [12:10:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1c5105770151...058982d9335e [12:10:22] [url] Comparing 1c5105770151...058982d9335e · miraheze/ssl · GitHub | github.com [12:10:23] [02miraheze/ssl] 07MirahezeSSLBot 03058982d - Bot: Update SSL cert for notes.mridulpm.in [12:11:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/22b64c7bd8d2...ba170d0e9a20 [12:11:04] [url] Comparing 22b64c7bd8d2...ba170d0e9a20 · miraheze/puppet · GitHub | github.com [12:11:05] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Sat 24 Sep 2022 13:22:32 GMT +0000. [12:11:05] [02miraheze/puppet] 07paladox 03ba170d0 - Fix [12:11:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 3.60, 4.06, 3.51 [12:11:45] PROBLEM - cp21 Stunnel HTTP for mon141 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.27, 3.59, 3.41 [12:14:01] RECOVERY - cp21 Stunnel HTTP for mon141 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37531 bytes in 2.062 second response time [12:14:18] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ba170d0e9a20...d7d6ce8df0bb [12:15:06] [url] Comparing ba170d0e9a20...d7d6ce8df0bb · miraheze/puppet · GitHub | github.com [12:15:07] [02miraheze/puppet] 07paladox 03d7d6ce8 - varnish::stunnel4: fix check for (cp31 Stunnel HTTP for reports121) [12:15:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.10, 3.87, 3.54 [12:16:56] PROBLEM - wiki.digitalcandela.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.15, 3.65, 3.50 [12:19:22] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.97, 5.07, 4.04 [12:20:34] RECOVERY - cp30 Stunnel HTTP for mon141 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 37546 bytes in 0.383 second response time [12:20:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [12:21:01] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.72, 1.60, 1.41 [12:21:04] RECOVERY - cp30 Stunnel HTTP for matomo131 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.817 second response time [12:21:11] RECOVERY - cp31 Stunnel HTTP for mon141 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 37538 bytes in 0.350 second response time [12:21:27] RECOVERY - cp31 Stunnel HTTP for matomo131 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 2.734 second response time [12:22:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d7d6ce8df0bb...d42d4dcd9218 [12:22:15] [url] Comparing d7d6ce8df0bb...d42d4dcd9218 · miraheze/puppet · GitHub | github.com [12:22:17] [02miraheze/puppet] 07paladox 03d42d4dc - Fix [12:22:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [12:23:46] RECOVERY - commons.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.gyaanipedia.com' will expire on Mon 21 Nov 2022 06:24:36 GMT +0000. [12:24:39] PROBLEM - wiki.alathramc.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.alathramc.com' expires in 15 day(s) (Tue 20 Sep 2022 12:13:47 GMT +0000). [12:24:58] RECOVERY - cp31 Stunnel HTTP for reports121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.266 second response time [12:25:14] RECOVERY - cp30 Stunnel HTTP for reports121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.288 second response time [12:26:53] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.19, 1.91, 1.60 [12:28:50] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.57, 1.77, 1.59 [12:30:47] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.46, 1.68, 1.58 [12:30:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/058982d9335e...450a1e3f9113 [12:30:49] [url] Comparing 058982d9335e...450a1e3f9113 · miraheze/ssl · GitHub | github.com [12:30:50] [02miraheze/ssl] 07MirahezeSSLBot 03450a1e3 - Bot: Update SSL cert for wiki.alathramc.com [12:36:32] RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:36:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6f7d98d6a970 [12:36:52] [02miraheze/puppet] 07paladox 036f7d98d - varnish::stunnel4: simplify check [12:36:53] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [12:36:54] [url] Page not found · GitHub · GitHub | github.com [12:36:55] [02puppet] 07paladox opened pull request 03#2804: varnish::stunnel4: simplify check - 13https://github.com/miraheze/puppet/pull/2804 [12:36:55] [url] Page not found · GitHub · GitHub | github.com [12:36:58] RECOVERY - notes.mridulpm.in - LetsEncrypt on sslhost is OK: OK - Certificate 'notes.mridulpm.in' will expire on Sat 03 Dec 2022 11:10:16 GMT +0000. [12:37:39] RECOVERY - test1.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [12:39:15] [02puppet] 07paladox closed pull request 03#2804: varnish::stunnel4: simplify check - 13https://github.com/miraheze/puppet/pull/2804 [12:39:15] [url] Page not found · GitHub · GitHub | github.com [12:39:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [12:39:18] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [12:39:18] [url] Page not found · GitHub · GitHub | github.com [12:39:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d42d4dcd9218...4c01d0ed9491 [12:39:20] [url] Comparing d42d4dcd9218...4c01d0ed9491 · miraheze/puppet · GitHub | github.com [12:39:21] [02miraheze/puppet] 07paladox 034c01d0e - varnish::stunnel4: simplify check (#2804) [12:43:22] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.81, 3.34, 3.84 [12:43:33]