[00:00:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.03, 1.74, 1.17 [00:01:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.61, 10.96, 10.50 [00:02:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.22, 9.87, 9.08 [00:02:33] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.48, 1.39, 1.11 [00:03:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.22, 2.09, 1.39 [00:03:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.80, 11.11, 10.63 [00:03:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.57, 11.15, 10.63 [00:04:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.21, 9.35, 9.00 [00:05:27] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.94, 1.42 [00:05:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 10.67, 10.54 [00:06:30] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:32] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:39] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:41] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.93, 9.19, 9.96 [00:08:29] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.045 second response time [00:08:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.966 second response time [00:08:43] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.985 second response time [00:08:49] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.158 second response time [00:09:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.76, 1.60, 1.41 [00:10:49] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.132 second response time [00:11:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.92, 9.50, 9.20 [00:11:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.66, 11.24, 10.77 [00:12:15] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.78, 3.40, 2.30 [00:13:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.51, 8.98, 9.04 [00:13:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 10.92, 10.70 [00:14:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:08] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:12] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:15] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:15] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.72, 3.26, 2.51 [00:16:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 10.76, 10.76 [00:16:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 4.448 second response time [00:17:19] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.504 second response time [00:17:22] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.032 second response time [00:19:09] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.557 second response time [00:19:11] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.056 second response time [00:20:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 11.13, 10.90 [00:22:07] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:16] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.08, 9.22, 10.00 [00:24:02] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.707 second response time [00:24:12] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.373 second response time [00:25:53] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.69, 9.89, 9.31 [00:27:58] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.085 second response time [00:28:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 11.74, 11.14 [00:29:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.20, 10.67, 10.37 [00:29:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.04, 10.35, 9.53 [00:31:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 11.36, 10.66 [00:33:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.51, 11.72, 10.36 [00:37:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.46, 11.88, 11.16 [00:40:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.47, 10.37, 9.49 [00:40:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 11.35, 11.50 [00:41:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.56, 9.75, 9.91 [00:42:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.73, 10.03, 9.47 [00:44:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.37, 12.34, 11.84 [00:45:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.68, 11.22, 10.97 [00:46:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.75, 10.79, 9.84 [00:47:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 10.59, 10.78 [00:48:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 11.36, 11.58 [00:49:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.31, 5.75, 4.09 [00:50:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 10.71, 10.08 [00:51:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 8.88, 10.03 [00:53:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.91, 10.74, 10.22 [00:55:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.28, 9.76, 9.92 [00:56:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.06, 10.04, 10.04 [00:59:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.31, 6.63, 5.83 [01:01:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.22, 1.66, 1.99 [01:04:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.62, 10.65, 10.13 [01:06:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.72, 9.89, 9.91 [01:13:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.82, 1.86 [01:15:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.83, 1.85 [01:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.55, 2.28, 2.00 [01:18:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.96, 11.39, 10.84 [01:22:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 11.55, 11.03 [01:25:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.69, 10.13, 9.88 [01:28:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.90, 10.36, 9.95 [01:28:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.45, 11.09 [01:29:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.90, 2.00 [01:30:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.21, 11.13, 11.01 [01:33:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.33, 9.49, 9.79 [01:33:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.94, 10.77, 9.84 [01:33:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 2.09, 2.06 [01:33:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.48, 11.27, 10.43 [01:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 11.92, 11.33 [01:35:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 10.70, 10.34 [01:37:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.39, 9.97, 9.75 [01:37:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.77, 9.84, 10.05 [01:42:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.08, 11.81, 11.68 [01:49:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.76, 10.83, 10.38 [01:52:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.58, 1.32 [01:53:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.45, 11.19, 10.59 [01:54:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.35, 1.76, 1.41 [01:55:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 11.14, 10.66 [01:56:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.53, 1.69, 1.43 [02:01:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 11.91, 11.11 [02:02:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.82, 11.44, 10.25 [02:02:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.99, 11.62, 11.37 [02:03:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.64, 11.87, 11.20 [02:04:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 10.51, 10.07 [02:04:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.45, 11.34 [02:06:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.53, 9.92, 9.91 [02:09:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:24] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.710 second response time [02:11:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 10.88, 10.84 [02:13:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 10.49, 10.71 [02:17:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [02:20:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.14, 11.00 [02:22:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.92, 11.32, 11.08 [02:26:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.27, 12.16, 11.44 [02:31:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 9.35, 10.02 [02:32:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.58, 11.88, 11.55 [02:43:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.79, 10.47, 9.91 [02:43:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.66, 1.91 [02:45:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.80, 9.56, 9.65 [02:47:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [02:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 2.05, 1.99 [02:57:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.87, 1.98 [02:57:47] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.72, 3.24, 2.74 [02:58:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.86, 11.63, 11.12 [02:59:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.89, 1.98 [03:00:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 11.03, 10.96 [03:01:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.80, 1.94 [03:03:33] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.05, 2.76, 2.70 [03:04:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.49, 10.89, 9.79 [03:06:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.70, 9.93, 9.57 [03:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 10.93, 10.85 [03:09:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.88, 1.91 [03:10:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.79, 11.16, 10.98 [03:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.82, 1.89 [03:12:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.74, 10.20, 9.71 [03:14:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.59, 9.23, 9.41 [03:15:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.88, 1.90 [03:18:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [03:30:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.67, 9.09, 10.09 [03:34:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.79, 2.41, 1.64 [03:36:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.88, 10.45, 10.39 [03:38:37] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.61, 1.49 [03:42:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 9.61, 10.10 [03:48:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:51:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.03, 10.37, 10.30 [04:00:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.50, 9.72, 10.07 [04:04:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 10.63, 10.35 [04:06:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.87, 10.05, 9.43 [04:06:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 11.68, 10.78 [04:10:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 11.24, 10.82 [04:11:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.83, 9.85, 9.58 [04:16:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.35, 10.95 [04:20:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.29, 11.80, 11.25 [04:31:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.64, 1.95 [04:32:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.35, 9.27, 10.16 [04:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.38, 10.42 [04:39:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.66, 1.80 [04:40:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.01, 11.22, 10.71 [04:41:12] PROBLEM - wiki.digitalcandela.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.digitalcandela.com' expires in 15 day(s) (Thu 04 Aug 2022 04:27:11 GMT +0000). [04:41:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.73, 1.82 [04:42:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 11.20, 10.77 [04:42:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [04:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.20, 2.23, 1.98 [04:47:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [04:48:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c98ee6f176a...13156c5f9014 [04:48:39] [url] Comparing 4c98ee6f176a...13156c5f9014 · miraheze/ssl · GitHub | github.com [04:48:40] [02miraheze/ssl] 07MirahezeSSLBot 0313156c5 - Bot: Update SSL cert for wiki.digitalcandela.com [05:00:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.78, 9.28, 10.01 [05:08:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 10.52, 10.09 [05:09:54] RECOVERY - wiki.digitalcandela.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitalcandela.com' will expire on Mon 17 Oct 2022 03:48:33 GMT +0000. [05:10:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 10.35, 10.08 [05:12:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.73, 10.24 [05:16:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.14, 10.52 [05:21:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.63, 1.85, 1.38 [05:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 10.75, 10.44 [05:23:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.72, 1.39 [05:24:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [05:24:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.83, 1.44, 1.33 [05:26:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 10.87, 10.58 [05:30:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.41, 10.24, 9.25 [05:31:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.76, 10.24, 9.57 [05:32:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.81, 9.56, 9.12 [05:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 10.95, 10.69 [05:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.38, 10.90 [05:38:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 9.81, 9.71 [05:48:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 11.00, 10.79 [05:50:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.37, 10.90, 10.79 [05:54:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 10.93, 10.78 [05:54:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [05:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 11.46, 11.06 [06:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.41, 11.70, 11.19 [06:06:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 11.67, 11.25 [06:32:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.34, 9.20, 8.48 [06:34:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.10, 8.86, 8.45 [06:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.29, 11.37, 10.83 [06:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.75, 11.78, 11.07 [06:44:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.44, 9.52, 10.15 [06:50:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 10.06, 10.11 [06:58:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.74, 11.69, 10.78 [07:00:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 11.17, 10.71 [07:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 11.86, 11.07 [07:08:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.41, 11.49, 11.05 [07:16:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.04, 9.49, 10.20 [07:20:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.68, 10.05, 10.31 [07:22:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 9.06, 9.92 [07:28:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 9.99, 10.05 [07:30:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.31, 9.85, 10.00 [07:45:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.26, 10.12, 9.99 [07:47:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.02, 9.62, 9.82 [07:58:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 10.51, 10.08 [08:01:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.21, 9.72, 8.82 [08:02:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 9.55, 9.83 [08:03:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.39, 9.03, 8.66 [08:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.62, 3.94 [08:36:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.98, 4.04 [08:38:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.55, 3.88 [08:42:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 4.14, 4.06 [08:50:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.37, 11.45, 10.08 [08:52:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.19, 11.35, 10.21 [09:00:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.42, 10.09, 10.10 [09:03:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 9.56, 8.84 [09:04:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 10.82, 10.40 [09:04:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.23, 1.86, 1.47 [09:05:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.95, 9.43, 8.89 [09:05:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.52, 10.23, 9.35 [09:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 11.83, 10.83 [09:06:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.14, 1.61, 1.43 [09:07:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.42, 9.92, 9.35 [09:08:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 11.47, 10.82 [09:09:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [09:11:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.19, 10.88, 9.90 [09:12:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.28, 3.79 [09:12:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.53, 10.52, 9.51 [09:14:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.86, 3.93 [09:14:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.61, 10.13, 9.49 [09:15:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.46, 9.88, 9.74 [09:15:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.82, 1.73, 1.18 [09:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.59, 3.81 [09:16:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.15, 3.84 [09:17:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.23, 1.46, 1.14 [09:18:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.87, 4.28, 4.02 [09:18:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.41, 3.84 [09:19:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [09:22:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.91, 3.95 [09:24:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.57, 3.83 [09:26:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.26, 5.03, 4.34 [09:26:31] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.06, 1.94, 1.53 [09:26:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.94, 3.93 [09:28:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.90, 1.47, 1.40 [09:30:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.74, 9.77, 10.17 [09:46:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.77, 4.00 [09:48:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.89, 4.22, 4.17 [10:01:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.72, 1.97 [10:04:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.14, 9.67 [10:08:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.89, 9.33, 9.50 [10:15:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.98, 1.95 [10:20:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 9.61, 9.31 [10:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 10.48, 9.67 [10:26:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.20, 10.26, 9.79 [10:28:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.82, 10.11, 9.79 [10:32:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 10.36, 10.00 [10:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.49, 3.96 [10:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 10.89, 10.23 [10:36:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 3.96, 4.06 [10:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 10.84, 10.29 [10:40:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.66, 3.94 [10:40:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.93, 10.09, 10.08 [10:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 4.29, 4.08 [11:06:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 10.37, 10.04 [11:22:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.69, 3.99 [11:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.68, 4.53, 4.22 [11:29:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.02, 9.68, 10.15 [11:29:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.32, 1.66, 1.96 [11:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.72, 3.98 [11:36:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.46, 4.49, 4.23 [11:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.00, 2.32, 2.00 [11:50:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.67, 2.17, 1.46 [11:52:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.75, 1.67, 1.36 [12:01:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.70, 10.69, 10.27 [12:02:31] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.96, 2.56, 1.75 [12:04:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.85, 1.95, 1.62 [12:05:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.72, 11.94, 10.84 [12:08:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.90, 1.61, 1.58 [12:09:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.89, 11.44, 10.91 [12:12:38] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:16:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 11.62, 11.14 [12:18:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 11.51, 11.14 [12:20:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.37, 12.06, 11.38 [12:26:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.75, 11.86, 11.54 [12:28:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.80, 12.39, 11.76 [12:29:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.56, 11.19, 9.91 [12:30:04] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:33:34] PROBLEM - orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:33:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.71, 11.56, 10.29 [12:35:03] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.83, 11.22, 10.35 [12:38:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.01, 11.96, 11.90 [12:39:34] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:40:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 12.20, 12.00 [12:41:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.56, 1.19 [12:43:45] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.43, 1.84, 1.33 [12:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 11.64, 11.83 [12:45:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.76, 9.97, 10.17 [12:45:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.09, 1.50, 1.26 [12:46:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.87, 12.43, 12.09 [12:49:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.67, 1.38 [12:50:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 11.45, 11.78 [12:51:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.51, 1.24, 1.26 [12:52:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.61, 12.03, 11.93 [13:06:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 10.59, 10.21 [13:09:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.27, 1.61, 1.92 [13:10:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.62, 9.77, 10.01 [13:16:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 10.35, 10.17 [13:16:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.82, 10.56, 11.79 [13:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.97, 1.93, 1.87 [13:20:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.57, 9.96, 9.22 [13:22:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.86, 9.70, 9.23 [13:23:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.18, 9.89, 10.10 [13:25:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 11.27, 11.57 [13:27:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.27, 1.90, 2.00 [13:28:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.17, 10.83, 11.38 [13:30:51] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:49] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.760 second response time [13:32:52] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 5.887 second response time [13:36:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.10, 11.11 [13:38:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.32, 11.09, 11.11 [13:41:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.85, 9.82, 8.91 [13:42:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.52, 11.21, 9.86 [13:42:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.39, 11.21 [13:43:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.94, 1.92 [13:44:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 12.00, 11.33, 10.05 [13:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.10, 10.85, 11.03 [13:45:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.73, 9.70, 9.11 [13:49:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.39, 10.72, 10.05 [13:50:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.76, 12.14, 11.50 [13:52:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 11.76, 11.45 [13:53:22] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.72, 10.00, 9.94 [13:54:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.27, 9.81, 9.88 [13:54:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 12.60, 11.81 [13:56:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.73, 11.90, 11.65 [13:58:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.03, 10.30, 10.11 [13:59:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.28, 9.74, 9.74 [14:00:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.89, 10.00, 10.04 [14:01:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.83, 9.29, 9.57 [14:10:03] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:14:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.61, 2.48, 1.68 [14:16:36] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.16, 1.96, 1.58 [14:18:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.66, 1.51, 1.46 [14:18:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.01, 10.16, 9.64 [14:20:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.50, 9.68, 9.52 [14:24:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.07, 10.72 [14:26:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.11, 10.78 [14:28:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 11.52, 10.97 [14:28:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.68, 10.50, 9.82 [14:30:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 11.32, 10.98 [14:31:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.18, 10.31, 9.88 [14:32:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.98, 9.60, 9.65 [14:32:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.72, 10.35, 9.75 [14:34:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.34, 10.04, 9.71 [14:39:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.03, 10.05, 10.02 [14:42:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.04, 10.38, 9.94 [14:46:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.76, 9.61, 9.73 [14:52:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.30, 11.03, 10.74 [14:54:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.78, 10.55, 10.60 [14:59:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.96, 9.70, 9.54 [14:59:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.62, 1.93 [15:01:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.80, 10.63, 9.89 [15:03:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 10.34, 9.86 [15:03:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.78, 1.92 [15:05:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.32, 10.04, 9.82 [15:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.28, 11.17, 10.77 [15:08:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.74, 10.83, 10.69 [15:09:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.80, 1.89 [15:12:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 11.19, 10.83 [15:14:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 11.49, 11.01 [15:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.86, 1.85 [15:19:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.71, 3.41, 3.19 [15:21:33] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.39, 3.12, 3.11 [15:26:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.62, 9.53, 10.20 [15:27:10] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406963586807251 seconds [15:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.20, 3.93 [15:42:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.62, 3.96 [15:47:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 1.71, 1.23 [15:48:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.11, 4.36, 4.02 [15:49:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.01, 1.41, 1.17 [15:50:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.90, 3.92 [15:53:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.57, 10.26, 9.87 [15:56:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.84, 3.96 [15:57:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 11.07, 10.27 [15:58:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.98, 4.64, 4.24 [15:58:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.46, 10.02, 9.12 [15:59:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.40, 10.80, 10.28 [16:00:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 9.79, 9.14 [16:04:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.73, 11.75, 10.74 [16:06:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.79, 11.26, 10.69 [16:08:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.66, 3.98 [16:10:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.11, 4.70, 2.63 [16:12:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.88, 3.98 [16:14:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.83, 3.96 [16:14:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.50, 3.96 [16:16:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 3.95, 3.98 [16:17:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:18:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.92, 2.00 [16:19:36] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.19, 1.78, 1.22 [16:21:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.52, 1.71, 1.26 [16:22:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [16:22:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.51, 3.82 [16:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.98, 10.67 [16:23:31] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.89, 1.39, 1.19 [16:24:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.64, 3.82 [16:26:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.36, 3.69 [16:26:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.07, 11.41, 10.95 [16:26:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.46, 3.62, 3.68 [16:26:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 10.51, 10.02 [16:27:16] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 4.04, 3.91 [16:28:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.26, 11.92, 11.18 [16:28:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.04, 1.33, 1.65 [16:28:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.83, 3.78 [16:30:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.68, 3.79 [16:30:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 11.68, 11.19 [16:30:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.68, 10.10, 9.96 [16:33:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.99, 3.42, 3.90 [16:37:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.35, 3.71 [16:37:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.64, 1.97 [16:39:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.03, 3.55 [16:40:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.97, 3.65, 3.59 [16:40:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.66, 3.57 [16:42:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.54, 3.56 [16:42:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.35, 9.47 [16:42:31] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [16:44:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.15, 9.85, 9.39 [16:45:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.32, 4.32, 3.85 [16:45:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.83, 10.36, 10.04 [16:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.81, 3.66 [16:46:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.89, 3.71 [16:47:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.93, 3.76 [16:47:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.91, 10.01, 9.96 [16:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.98, 1.92, 1.87 [16:48:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.52, 5.29, 4.21 [16:49:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.73, 5.14, 4.23 [16:50:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.74, 3.66 [16:52:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 10.23, 9.59 [16:54:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.47, 9.83, 9.52 [16:54:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.72, 3.83 [16:55:42] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.72, 3.45, 3.03 [16:57:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.85, 1.96 [16:57:38] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.39, 2.99, 2.91 [16:58:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.16, 11.26, 10.85 [16:58:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.33, 4.02 [16:58:51] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:08] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.68, 10.19, 9.69 [16:59:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.467 second response time [17:01:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.402 second response time [17:01:07] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.974 second response time [17:01:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.17, 9.86, 9.61 [17:01:37] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [17:02:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.65, 11.10, 10.96 [17:02:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.87, 3.93 [17:04:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.32, 4.33, 4.08 [17:06:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.53, 11.82, 11.20 [17:08:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.80, 3.64 [17:10:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.44, 3.53 [17:10:31] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:12:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.49, 11.37, 9.71 [17:12:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.07, 11.99, 11.62 [17:12:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.85, 3.25, 3.77 [17:13:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 2.01, 1.92 [17:13:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.48, 9.25, 8.59 [17:15:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.26, 9.00, 8.56 [17:16:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.41, 10.20, 9.66 [17:19:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.60, 3.91 [17:20:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.07, 3.99, 3.89 [17:22:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.80, 3.84 [17:26:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.26, 3.40 [17:26:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.87, 4.09, 3.94 [17:28:53] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.08, 2.44, 1.57 [17:29:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.00, 4.31, 3.93 [17:30:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.92, 4.12, 3.76 [17:30:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.93, 9.72, 9.31 [17:32:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.00, 9.64, 9.33 [17:32:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.93, 1.70, 1.48 [17:33:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.57, 3.77, 3.82 [17:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.52, 10.65 [17:36:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.78, 3.98 [17:37:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 4.16, 3.96 [17:38:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.70, 11.83, 11.22 [17:38:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 4.09, 4.07 [17:40:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.33, 10.18, 9.50 [17:41:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.77, 3.86 [17:43:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.32, 4.41, 4.06 [17:46:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.77, 10.06, 9.64 [17:47:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.04, 9.67, 9.55 [17:48:20] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 8.93, 3.27, 2.04 [17:48:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.70, 3.96 [17:50:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.04, 9.85, 9.86 [17:50:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.23, 5.17, 4.45 [17:50:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.67, 9.24, 8.30 [17:52:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.41, 8.91, 8.29 [17:54:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.38, 1.97, 1.83 [17:56:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.68, 11.39, 10.97 [17:58:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.40, 1.62 [17:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.79, 11.55, 11.09 [18:00:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 11.78, 11.22 [18:01:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.23, 11.32, 10.04 [18:02:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.86, 11.17, 11.08 [18:04:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 11.01, 10.29 [18:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.87, 11.54, 11.20 [18:05:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.72, 11.97, 10.63 [18:06:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.45, 10.29, 10.11 [18:08:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.07, 9.78, 9.97 [18:12:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 10.29, 10.10 [18:12:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 11.78, 11.64 [18:13:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.70, 11.29, 10.67 [18:14:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.35, 9.04, 9.66 [18:14:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 12.41, 11.89 [18:14:38] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.98, 2.05, 1.54 [18:16:39] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.242 second response time [18:17:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.91, 11.50, 10.92 [18:18:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.76, 1.89, 1.54 [18:20:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.19, 1.61, 1.48 [18:24:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.44, 2.02, 1.68 [18:25:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.04, 9.15, 10.02 [18:26:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.58, 1.82, 1.65 [18:28:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.60, 1.59 [18:39:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [18:44:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.48, 10.44, 9.69 [18:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.58, 10.92, 11.74 [18:46:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.76, 9.79, 9.54 [18:50:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.49, 11.65 [18:53:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 1.86, 1.44 [18:54:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.32, 1.60, 1.39 [18:58:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.51, 4.00 [19:00:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.20, 4.24, 4.21 [19:06:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.29, 1.65, 1.43 [19:12:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.70, 10.36, 9.40 [19:14:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.41, 10.01, 9.39 [19:17:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.97, 10.12, 9.34 [19:18:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.98, 11.54, 11.95 [19:18:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 9.89, 9.34 [19:19:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.42, 9.55, 9.23 [19:20:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.25, 1.84, 1.78 [19:22:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.21, 9.68, 9.41 [19:24:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.68, 12.20, 12.04 [19:24:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.39, 1.62 [19:28:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 11.62, 11.84 [19:29:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.41, 10.54, 9.81 [19:31:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 10.39, 9.85 [19:33:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.40, 11.43, 10.20 [19:33:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.37, 10.21, 9.74 [19:34:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.83, 11.10, 10.21 [19:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.89, 11.32, 10.33 [19:36:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 11.68, 11.75 [19:36:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.76, 10.47, 10.08 [19:37:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.66, 11.90, 10.66 [19:37:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.23, 9.82, 9.73 [19:40:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 11.79, 11.82 [19:40:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 10.99, 10.36 [19:41:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.24, 11.58, 10.84 [19:42:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.80, 10.36, 9.39 [19:42:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.90, 11.25, 10.53 [19:43:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.68, 1.76, 1.59 [19:44:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.50, 9.26, 9.10 [19:45:17] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.68, 1.66, 1.57 [19:45:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.45, 11.18, 10.39 [19:47:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.53, 11.34, 10.85 [19:47:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.11, 10.80, 10.34 [19:48:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 11.70, 11.73 [19:49:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.57, 11.36, 10.92 [19:49:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.23, 9.71, 9.99 [19:57:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.47, 11.71, 11.18 [19:59:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.88, 11.73, 11.25 [20:00:55] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.85, 3.38, 3.02 [20:02:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 11.48, 11.85 [20:02:55] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.07, 3.36, 3.06 [20:03:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.45, 11.54, 11.20 [20:03:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.51, 10.58, 10.21 [20:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 12.15, 12.05 [20:05:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.16, 11.18, 11.13 [20:08:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.82, 9.95, 10.19 [20:09:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.71, 10.03, 9.41 [20:11:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.24, 10.09, 10.16 [20:11:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.40, 9.78, 9.40 [20:12:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.12, 11.47, 10.68 [20:12:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 11.59, 11.89 [20:14:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.53, 10.63, 10.47 [20:16:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.96, 9.55, 10.11 [20:16:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 11.73, 11.85 [20:17:44] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.78, 9.12, 10.08 [20:20:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 11.65, 11.76 [20:22:29] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.37, 12.20, 11.94 [20:24:33] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.770 second response time [20:24:51] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.859 second response time [20:25:57] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.016 second response time [20:27:00] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.538 second response time [20:27:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.420 second response time [20:27:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.82, 10.09, 9.90 [20:29:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.49, 9.88, 9.85 [20:29:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 10.43, 10.05 [20:31:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.32, 10.15, 10.01 [20:33:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.05, 10.35, 10.09 [20:34:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.11, 9.85, 9.48 [20:35:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.06, 9.55, 9.83 [20:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.78, 10.88, 10.35 [20:36:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.97, 9.68, 9.45 [20:37:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.69, 11.52, 10.65 [20:38:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.10, 11.00, 11.68 [20:39:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.54, 10.79, 10.49 [20:40:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.20, 10.59, 9.90 [20:43:56] PROBLEM - theapplesecretswiki.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theapplesecretswiki.ml' expires in 15 day(s) (Thu 04 Aug 2022 20:37:27 GMT +0000). [20:45:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.31, 10.42, 9.76 [20:46:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 10.72, 11.12 [20:47:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.06, 10.00, 9.66 [20:49:12] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13156c5f9014...657606b64fbb [20:49:13] [url] Comparing 13156c5f9014...657606b64fbb · miraheze/ssl · GitHub | github.com [20:49:14] [02miraheze/ssl] 07MirahezeSSLBot 03657606b - Bot: Update SSL cert for theapplesecretswiki.ml [20:49:42] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.22, 3.37, 2.98 [20:50:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.23, 10.01, 9.92 [20:50:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 11.12, 11.20 [20:51:38] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.29, 3.29, 2.99 [20:51:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.34, 9.69, 10.15 [20:52:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.62, 11.37 [20:52:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.03, 2.28, 1.55 [20:58:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.48, 1.84, 1.60 [20:58:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.29, 10.13, 9.99 [21:00:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.65, 1.56 [21:02:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 11.93, 11.81 [21:03:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 10.80, 9.86 [21:03:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.10, 10.91, 10.40 [21:05:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.93, 9.81, 9.61 [21:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 12.14, 11.89 [21:07:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.67, 11.06, 10.63 [21:08:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.33, 10.07, 10.09 [21:10:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.34, 11.56, 11.76 [21:12:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.83, 10.83, 10.37 [21:14:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.43, 9.61, 9.99 [21:14:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [21:18:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.52, 11.75, 11.60 [21:20:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.14, 10.99, 10.12 [21:20:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.82, 11.67, 11.58 [21:21:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 11.50, 10.90 [21:22:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.10, 10.82, 10.29 [21:22:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.62, 11.11, 10.27 [21:23:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.65, 11.34, 10.93 [21:24:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.22, 10.92, 10.40 [21:24:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.54, 11.48, 10.50 [21:25:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.84, 11.81, 11.14 [21:28:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 10.81, 9.75 [21:29:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.51, 11.79, 11.28 [21:30:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.69, 11.41, 10.86 [21:30:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.21, 10.10, 9.62 [21:32:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.07, 11.95, 11.14 [21:33:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.50, 11.98, 11.45 [21:34:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.14, 9.43, 10.00 [21:34:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.98, 10.93, 10.87 [21:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.72, 12.16, 11.60 [21:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.21, 11.76, 11.44 [21:38:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.71, 3.97 [21:38:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.36, 3.95 [21:40:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 12.00, 10.38, 10.14 [21:40:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.09, 11.56, 11.64 [21:40:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.93, 3.96, 4.10 [21:42:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 12.11, 11.83 [21:43:48] RECOVERY - theapplesecretswiki.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'theapplesecretswiki.ml' will expire on Mon 17 Oct 2022 19:49:07 GMT +0000. [21:44:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.60, 9.86, 10.04 [21:44:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.50, 4.16, 4.04 [21:44:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.05, 10.78, 10.58 [21:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.78, 11.49, 11.65 [21:45:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:46:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.65, 10.44, 10.49 [21:46:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.30, 11.73, 11.71 [21:48:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.42, 9.67, 10.19 [21:49:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.26, 4.52, 2.66 [21:50:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.80, 10.94, 9.62 [21:50:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.60, 10.63, 10.30 [21:52:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.83, 10.66, 10.47 [21:52:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.69, 10.76, 9.70 [21:54:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 9.74, 9.47 [21:54:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.55, 9.90, 10.13 [21:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.75, 11.87, 11.94 [22:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.86, 3.99 [22:00:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.74, 3.96 [22:00:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.48, 1.95 [22:03:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.55, 11.13, 11.08 [22:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 11.54, 11.69 [22:04:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.92, 3.97 [22:04:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.01, 1.64 [22:05:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 11.62, 11.29 [22:06:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 10.77, 11.41 [22:06:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.53, 3.81 [22:08:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.78, 3.87 [22:10:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.46, 3.63 [22:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.74, 1.97 [22:12:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.44, 3.60 [22:12:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.83, 3.86 [22:13:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.63, 9.17, 10.19 [22:14:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 3.99, 3.78 [22:14:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.94, 9.65, 10.10 [22:14:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.04, 8.29, 9.89 [22:14:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.05, 3.92 [22:15:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:16:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.94, 3.79 [22:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.88, 2.43, 2.16 [22:18:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 4.39, 3.98 [22:22:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.30, 10.17, 9.43 [22:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 10.13, 10.04 [22:24:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.86, 9.55, 9.29 [22:24:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.25, 10.37, 10.14 [22:24:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.92, 3.99 [22:26:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.87, 3.96 [22:26:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.83, 3.96, 3.98 [22:28:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 4.29, 4.11 [22:28:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.29, 10.06, 10.09 [22:32:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.66, 3.90 [22:34:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.52, 10.40, 9.97 [22:36:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.65, 3.81 [22:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 11.19, 10.61 [22:36:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.03, 10.18, 9.93 [22:37:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.30, 3.93 [22:38:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.49, 3.73 [22:38:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.55, 3.88 [22:42:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 10.83, 10.57 [22:43:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.64, 3.86 [22:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.74, 10.58, 10.50 [22:45:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [22:46:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 11.09, 10.69 [22:48:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.08, 4.13, 3.86 [22:48:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 10.65, 10.59 [22:48:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.73, 3.83 [22:50:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.52, 3.67 [22:50:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.04, 3.10, 3.59 [22:51:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.80, 3.97 [22:53:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.24, 4.12 [22:54:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.03, 3.82 [22:55:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.77, 3.96 [22:55:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:56:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.84, 3.77 [22:56:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.80, 3.77 [23:01:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 4.34, 4.10 [23:02:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.92, 3.87 [23:02:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 9.94, 9.42 [23:04:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.96, 9.57, 9.35 [23:10:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.71, 3.62 [23:10:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.27, 9.02, 9.93 [23:10:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.81, 3.80 [23:10:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.40, 10.10, 9.54 [23:12:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.69, 3.63 [23:14:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.70, 9.74, 9.51 [23:16:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.73, 3.85 [23:17:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.13, 3.78 [23:18:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.80, 3.13, 3.39 [23:21:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.17, 10.42, 9.78 [23:22:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.95, 4.67, 4.13 [23:23:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.20, 3.96 [23:23:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 10.69, 10.31 [23:24:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.95, 3.93 [23:25:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.42, 4.54, 3.84 [23:25:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 9.59, 9.61 [23:26:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.89, 4.59, 4.16 [23:27:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.89, 3.90 [23:29:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 4.27, 4.04 [23:31:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.61, 3.83 [23:32:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.89, 3.93 [23:33:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.91, 3.90 [23:34:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.74, 3.97 [23:35:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.69, 3.81 [23:36:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.76, 3.95 [23:37:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.33, 4.84, 4.18 [23:37:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.61, 1.92 [23:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.80, 9.77, 10.13 [23:38:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.88, 3.85 [23:42:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.69, 3.78 [23:44:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 10.72, 10.21 [23:46:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.12, 9.50, 9.84 [23:46:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.15, 4.43, 4.02 [23:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.88, 2.01, 1.89 [23:52:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 10.69, 10.21 [23:54:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.49, 9.81, 9.95 [23:55:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.98, 2.00 [23:57:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.21, 9.74, 9.39 [23:59:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 3.40, 3.92 [23:59:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.34, 9.29, 9.28