[00:02:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.65, 2.04, 1.58 [00:02:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 11.42, 11.11 [00:03:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.74, 1.43 [00:04:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.71, 1.52 [00:04:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.11, 10.09, 9.12 [00:04:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 11.11, 11.04 [00:05:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.47, 1.63, 1.42 [00:06:09] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 4.00, 3.05, 2.26 [00:06:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.44, 1.44 [00:06:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.30, 9.13, 8.90 [00:08:10] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.97, 3.10, 2.38 [00:21:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:04] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:32] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 11.57, 10.86 [00:27:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [00:29:26] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.871 second response time [00:29:54] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.780 second response time [00:29:55] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28496 bytes in 7.800 second response time [00:33:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:32] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:34] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:57] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.766 second response time [00:39:37] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28496 bytes in 9.784 second response time [00:39:41] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.831 second response time [00:39:56] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.167 second response time [00:40:48] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.195 second response time [00:42:01] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.80, 3.37, 2.44 [00:43:17] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:00] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:05] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:15] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:18] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.588 second response time [00:46:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28496 bytes in 5.532 second response time [00:46:08] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.139 second response time [00:46:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.949 second response time [00:50:01] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.96, 3.99, 3.17 [00:52:01] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.77, 4.33, 3.40 [00:55:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:41] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:46] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:53] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.522 second response time [00:59:59] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.812 second response time [01:00:37] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28496 bytes in 1.664 second response time [01:00:41] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.105 second response time [01:00:53] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.725 second response time [01:02:01] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.92, 3.81, 3.73 [01:03:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [01:10:01] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.23, 2.82, 3.29 [01:13:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [01:14:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 10.41, 11.77 [01:18:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 11.30, 11.81 [01:30:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 11.41, 11.86 [01:34:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.80, 11.72, 11.84 [01:40:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.51, 1.28 [01:40:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.33, 11.77, 11.88 [01:42:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.64, 1.47, 1.28 [01:45:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.15, 1.75, 1.42 [01:47:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.63, 1.70, 1.44 [01:53:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.92, 2.25, 1.75 [01:54:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.41, 11.41 [01:56:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.82, 1.51 [01:56:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 11.60, 11.49 [01:57:34] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.79, 1.68 [01:57:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 9.68, 8.59 [01:58:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.75, 1.51 [01:58:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.35, 12.25, 11.75 [01:59:29] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.45, 1.56 [01:59:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.22, 9.03, 8.51 [02:02:24] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.51, 1.46 [02:12:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.39, 11.74, 11.96 [02:16:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 12.12, 12.05 [02:18:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 11.42, 11.79 [02:20:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.91, 11.94 [02:22:11] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.89, 1.73 [02:24:10] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.91, 1.75 [02:26:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.62, 1.66 [02:30:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.69, 1.72 [02:32:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.55, 1.66 [02:32:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 11.31, 11.85 [02:34:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 11.76, 11.93 [02:38:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 11.24, 11.75 [02:46:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.52, 3.02, 2.29 [02:48:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 9.54, 5.23, 2.98 [02:48:31] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:38] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:39] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:53] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:48:58] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:23] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:50:43] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.07, 1.50, 1.14 [02:52:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.30, 1.12 [02:52:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.69, 1.93 [02:54:36] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.75, 12.05, 11.57 [02:54:58] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:14] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:15] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:55:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:41] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.475 second response time [02:56:56] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.815 second response time [02:57:03] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.350 second response time [02:57:10] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.326 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [02:57:11] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:57:14] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.543 second response time [02:57:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [02:57:40] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28496 bytes in 5.628 second response time [02:58:42] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28504 bytes in 0.435 second response time [02:58:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.39, 1.70 [02:58:53] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.320 second response time [02:59:02] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.030 second response time [02:59:10] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.041 second response time [02:59:26] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.330 second response time [03:02:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:02:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 11.20, 11.62 [03:15:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.60, 1.51 [03:16:24] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.35, 0.65, 3.66 [03:17:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.87, 1.62 [03:18:18] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.21, 0.51, 3.26 [03:19:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.90, 1.67 [03:22:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.03, 10.99 [03:23:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 2.34, 1.88 [03:24:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.72, 10.89 [03:29:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.51, 1.93, 1.86 [03:33:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.68, 1.42, 1.68 [03:42:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.26, 8.88, 8.01 [03:44:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.68, 8.40, 7.94 [03:44:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.88, 9.56, 10.11 [04:23:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 9.79, 9.64 [04:25:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.99, 9.86, 9.69 [04:35:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.25, 2.19, 1.65 [04:37:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.83, 1.58 [04:39:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.50, 1.48 [04:54:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.49, 4.00 [04:55:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.37, 5.73, 3.86 [04:56:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.47, 3.55, 4.00 [04:57:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.67, 5.56, 4.02 [04:58:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.80, 4.04 [04:59:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.28, 6.16, 4.44 [05:00:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.88, 3.98 [05:03:24] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 4.48, 2.14, 1.10 [05:05:24] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 1.35, 1.73, 1.07 [05:07:24] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 1.25, 1.64, 1.12 [05:08:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.81, 4.00 [05:14:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.69, 3.97 [05:15:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.15, 5.28, 5.81 [05:18:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.80, 3.83 [05:18:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.76, 4.69, 4.26 [05:19:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.87, 5.35, 5.64 [05:22:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 4.00, 3.99 [05:24:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.07, 4.50, 4.17 [05:25:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.97, 5.40, 5.75 [05:27:00] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.46, 7.01, 6.28 [05:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.80, 3.97 [05:31:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.59, 3.34, 3.98 [05:32:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.27, 3.86 [05:39:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.61, 3.80 [05:41:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.23, 3.64 [05:42:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.01, 3.37 [05:42:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.82, 2.89, 3.38 [05:46:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.54, 3.56 [05:46:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.39, 3.50 [05:49:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.74, 3.69 [05:51:00] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.37, 4.37, 5.94 [05:51:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.71, 3.68 [05:52:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 10.79, 9.33 [05:52:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.42, 3.45 [05:54:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 3.09, 3.35 [05:54:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.83, 3.20, 3.37 [05:55:00] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.38, 2.50, 4.79 [05:59:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.98, 3.75 [05:59:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 11.78, 10.55 [06:00:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 4.16, 3.72 [06:00:37] PROBLEM - thenintendowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thenintendowiki.org' expires in 15 day(s) (Tue 09 Aug 2022 05:50:58 GMT +0000). [06:01:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.44, 12.45, 10.94 [06:06:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.88, 3.82 [06:06:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c22b51a76fd1...c2dbafa8a364 [06:06:30] [url] Comparing c22b51a76fd1...c2dbafa8a364 · miraheze/ssl · GitHub | github.com [06:06:31] [02miraheze/ssl] 07MirahezeSSLBot 03c2dbafa - Bot: Update SSL cert for thenintendowiki.org [06:07:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.55, 3.73 [06:11:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.77, 11.48 [06:11:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 2.74, 3.36 [06:12:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 2.80, 3.35 [06:17:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.97, 8.73, 10.16 [06:24:07] PROBLEM - www.scum-wiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'scum-wiki.com' expires in 15 day(s) (Tue 09 Aug 2022 06:16:18 GMT +0000). [06:27:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 10.97, 10.41 [06:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.01, 3.26 [06:29:15] RECOVERY - thenintendowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thenintendowiki.org' will expire on Sat 22 Oct 2022 05:06:23 GMT +0000. [06:29:16] PROBLEM - scum-wiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'scum-wiki.com' expires in 15 day(s) (Tue 09 Aug 2022 06:16:18 GMT +0000). [06:29:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.61, 3.15 [06:30:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.58, 3.06 [06:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.56, 3.18 [06:31:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 11.30, 10.71 [06:31:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 3.83, 3.28 [06:32:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 3.95, 3.25 [06:32:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.93, 3.35 [06:33:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.14, 11.93, 11.00 [06:33:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.64, 3.27 [06:34:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.28, 3.09 [06:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.71, 3.34 [06:35:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c2dbafa8a364...cfb973f525bf [06:35:15] [url] Comparing c2dbafa8a364...cfb973f525bf · miraheze/ssl · GitHub | github.com [06:35:16] [02miraheze/ssl] 07MirahezeSSLBot 03cfb973f - Bot: Update SSL cert for scum-wiki.com [06:37:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.24, 11.56, 11.19 [06:37:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.57, 3.30 [06:38:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 3.39, 3.29 [06:38:26] PROBLEM - smashbroswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'smashbroswiki.com' expires in 15 day(s) (Tue 09 Aug 2022 06:17:55 GMT +0000). [06:39:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 4.01, 3.54 [06:39:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.92, 3.32, 3.24 [06:42:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.93, 3.52 [06:43:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cfb973f525bf...0879ac84dfb2 [06:43:06] [url] Comparing cfb973f525bf...0879ac84dfb2 · miraheze/ssl · GitHub | github.com [06:43:07] [02miraheze/ssl] 07MirahezeSSLBot 030879ac8 - Bot: Update SSL cert for smashbroswiki.com [06:43:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.71, 9.25, 10.20 [06:43:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.11, 4.66, 3.81 [06:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.71, 3.48 [06:45:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.93, 3.71 [06:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.58, 3.45 [06:47:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.40, 3.32, 3.49 [06:48:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 3.00, 3.25 [06:51:04] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dkwiki.org' expires in 15 day(s) (Tue 09 Aug 2022 06:27:10 GMT +0000). [06:51:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 3.15, 3.38 [06:57:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.67, 3.49 [06:57:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.73, 3.62 [06:58:02] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [06:58:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0879ac84dfb2...9e60dc964999 [06:58:58] [url] Comparing 0879ac84dfb2...9e60dc964999 · miraheze/ssl · GitHub | github.com [06:58:59] [02miraheze/ssl] 07MirahezeSSLBot 039e60dc9 - Bot: Update SSL cert for dkwiki.org [07:00:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.63, 3.31 [07:01:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.54, 3.47 [07:01:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.34, 3.51, 3.59 [07:02:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.21, 3.19 [07:03:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.60, 3.49 [07:03:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.33, 4.09, 3.78 [07:05:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [07:05:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 3.22, 3.38 [07:05:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.72, 3.69 [07:07:17] RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Sat 22 Oct 2022 05:43:00 GMT +0000. [07:07:48] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:11:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.92, 3.76 [07:12:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.64, 3.57 [07:13:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.41, 3.58 [07:14:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.78, 3.61 [07:16:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.79, 3.64 [07:17:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.20, 2.90, 3.35 [07:19:37] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [07:21:49] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:23:02] PROBLEM - www.scum-wiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:47] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw101 mw102 [07:26:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.35, 2.88, 3.28 [07:28:45] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:30:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.98, 3.58 [07:32:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [07:32:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.43, 3.38 [07:33:44] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:33:49] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw101 [07:34:11] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:34:25] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 13.10, 6.27, 3.31 [07:34:31] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:39] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:34:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [07:34:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.83, 3.59 [07:34:58] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:22] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:53] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:36:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 3.91, 3.54 [07:36:45] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:37:46] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:38:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.71, 3.51 [07:39:43] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 15 minutes ago with 0 failures [07:39:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.83, 10.35, 9.37 [07:40:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [07:40:09] PROBLEM - sims.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:37] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28512 bytes in 3.998 second response time [07:40:43] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.337 second response time [07:41:11] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.339 second response time [07:41:37] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:41:41] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw102 mw121 [07:41:54] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 2.065 second response time [07:42:33] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 1.035 second response time [07:42:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.42, 3.85, 3.90 [07:43:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.37, 10.00, 9.47 [07:44:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:44:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.45, 4.45, 3.75 [07:46:46] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw121 mw122 [07:49:00] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [07:49:39] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [07:49:54] PROBLEM - erawikisource.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:14] PROBLEM - cp21 Stunnel HTTP for mon111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:21] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.07, 3.59, 3.66 [07:52:25] RECOVERY - www.scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Sat 22 Oct 2022 05:35:09 GMT +0000. [07:53:06] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [07:54:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.37, 3.58 [07:55:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw101 mw111 [07:56:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 2.92, 3.36 [07:56:32] RECOVERY - cp21 Stunnel HTTP for mon111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 37927 bytes in 7.139 second response time [07:56:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 2.89, 3.38 [07:58:27] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.96, 2.38, 3.99 [07:59:46] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:48] PROBLEM - squareenix.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:51] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:06] PROBLEM - celeste.ink - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:06] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.63, 9.61, 7.57 [08:03:50] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [08:04:11] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.65, 1.87, 3.27 [08:04:26] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:04:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.02, 9.25, 7.71 [08:08:57] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:09:01] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [08:10:14] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [08:15:57] PROBLEM - drowiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:49] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:06] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [08:19:08] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [08:19:20] RECOVERY - erawikisource.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:19:54] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 22 Oct 2022 05:58:53 GMT +0000. [08:20:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 12.08, 6.14, 3.93 [08:20:15] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:20:29] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:31] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:39] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:41] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:58] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:21:08] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [08:22:32] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [08:23:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [08:24:30] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28512 bytes in 0.500 second response time [08:24:33] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.020 second response time [08:24:39] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.336 second response time [08:24:49] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.035 second response time [08:25:03] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.310 second response time [08:25:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [08:25:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.77, 3.22 [08:27:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:27:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.26, 3.09 [08:28:06] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [08:28:33] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Wed 24 Aug 2022 04:49:04 GMT +0000. [08:28:39] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [08:30:30] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:30:48] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Wed 24 Aug 2022 02:57:37 GMT +0000. [08:31:46] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [08:32:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [08:34:01] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.34, 2.62, 3.99 [08:34:01] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [08:34:09] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Thu 29 Sep 2022 02:01:07 GMT +0000. [08:35:58] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [08:36:56] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [08:38:01] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.22, 1.82, 3.34 [08:40:49] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw102 mw112 mw121 [08:42:38] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:44:30] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 04 Oct 2022 18:56:56 GMT +0000. [08:44:48] PROBLEM - commonswiki.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:52] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [08:45:49] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw101 mw121 mw122 [08:49:45] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [08:52:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.67, 3.11 [08:54:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.30, 3.04 [08:56:47] PROBLEM - commons.ballmedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:16] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:00:21] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Thu 15 Sep 2022 05:33:41 GMT +0000. [09:04:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.89, 10.11, 8.40 [09:06:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.73, 10.66, 8.81 [09:07:37] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [09:08:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.86, 9.83, 8.72 [09:09:25] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [09:12:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [09:13:18] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw111 mw122 [09:13:42] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [09:14:05] PROBLEM - cp20 Stunnel HTTP for mon111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.10, 2.78 [09:15:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 3.76, 3.22 [09:15:33] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw111 mw121 [09:15:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.36, 3.05 [09:16:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 2.63, 2.64 [09:17:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.82, 3.31, 3.12 [09:17:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 2.99, 2.96 [09:20:23] RECOVERY - cp20 Stunnel HTTP for mon111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 37919 bytes in 0.049 second response time [09:23:36] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 15.21, 7.01, 3.82 [09:24:31] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:42] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:58] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:21] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:51] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:25:51] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:26:13] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:33] RECOVERY - commons.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.ballmedia.org' will expire on Mon 10 Oct 2022 06:16:04 GMT +0000. [09:26:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.76, 4.68, 3.74 [09:27:06] RECOVERY - grc.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grc.repository.archiopedia.org' will expire on Sat 17 Sep 2022 19:06:52 GMT +0000. [09:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.57, 3.27 [09:27:38] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.087 second response time [09:27:45] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.377 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::4c25,51.195.220.68 [09:27:50]