[00:01:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.82, 3.33, 3.24 [00:01:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 4.04, 4.11 [00:04:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.45, 4.82, 3.90 [00:05:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.93, 3.51 [00:10:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.87, 3.71 [00:14:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.55, 3.80 [00:16:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.59, 3.75 [00:18:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.61, 3.75 [00:23:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 2.22, 1.66 [00:24:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 2.90, 3.32 [00:25:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.86, 1.64, 1.52 [00:32:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 3.81, 3.68 [00:36:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.50, 3.72, 3.73 [00:40:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.24, 1.50, 1.99 [00:40:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.79, 3.72 [00:42:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.91, 11.29, 9.61 [00:42:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.52, 3.26, 3.53 [00:44:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.33, 9.84 [00:44:45] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.314 second response time [00:44:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.87, 3.36 [00:45:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.15, 10.12, 9.58 [00:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.71, 1.86 [00:51:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.71, 3.98 [00:52:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.89, 1.92 [00:57:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 10.49, 9.93 [00:57:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.79, 3.89 [00:58:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.12, 1.36, 1.67 [00:58:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.52, 9.76, 8.94 [00:59:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.50, 3.33 [00:59:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 10.13, 9.88 [00:59:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.80, 3.88 [01:00:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.75, 9.51, 8.96 [01:01:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 4.07, 3.96 [01:04:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 2.02, 1.82 [01:05:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.26, 10.36 [01:05:52] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.94, 1.82 [01:07:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.75, 3.47 [01:07:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 10.50, 10.21 [01:07:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23900 bytes in 0.907 second response time [01:08:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.00, 2.04, 1.87 [01:09:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.68, 3.48 [01:09:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.28, 9.94, 10.03 [01:10:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.86, 1.83 [01:10:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.80, 4.43, 3.77 [01:12:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.60, 3.30 [01:13:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.85, 3.97 [01:14:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.20, 4.20, 3.56 [01:16:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.69, 3.45 [01:17:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.17, 4.08 [01:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 1.94, 1.84 [01:18:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.83, 3.82 [01:19:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.83, 3.97 [01:20:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.17, 3.29 [01:21:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.38, 4.16 [01:22:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.72, 10.42, 10.12 [01:22:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 2.00, 1.91 [01:24:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.55, 10.97, 10.34 [01:25:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.55, 3.87 [01:26:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.83, 11.51, 10.63 [01:26:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 3.90, 3.83 [01:27:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 9.74, 9.24 [01:27:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.99, 3.98 [01:28:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.85, 3.82 [01:29:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.18, 9.68, 9.26 [01:29:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.71, 3.88 [01:30:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.42, 3.35 [01:30:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.02, 2.57, 2.13 [01:30:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.85, 3.81 [01:31:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 4.05, 3.98 [01:32:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.72, 3.77 [01:33:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.50, 9.61, 8.48 [01:34:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 10.48, 10.40 [01:34:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.14, 3.25 [01:35:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.59, 8.73, 8.30 [01:35:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.77, 3.89 [01:37:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.86, 10.64 [01:39:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 9.61, 10.20 [01:40:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 2.69, 3.25 [01:41:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.45, 3.64 [01:43:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.38, 3.60 [01:45:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.55, 11.22, 10.01 [01:45:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.77, 3.71 [01:45:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.01, 12.08, 10.90 [01:46:26] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.45, 11.58, 10.31 [01:47:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.72, 3.70 [01:48:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.335 second response time [01:49:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.83, 4.56, 3.73 [01:49:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 4.03, 3.46 [01:49:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 11.32, 10.91 [01:49:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.36, 3.94 [01:51:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.71, 11.84, 10.67 [01:51:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.37, 3.29 [01:51:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.15, 12.09, 11.24 [01:52:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.62, 1.27 [01:53:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.80, 11.21, 10.57 [01:53:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.76, 3.80 [01:54:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.54, 1.28 [01:55:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.95, 3.75 [01:55:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 10.97, 11.02 [01:55:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 3.92, 3.84 [01:57:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.89, 4.05, 3.82 [01:57:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.63, 3.40 [01:58:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.82, 1.46 [01:59:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.09, 9.53, 10.05 [01:59:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.37, 3.33 [01:59:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.52, 1.39 [02:02:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.89, 3.85 [02:03:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.00, 8.80, 9.96 [02:05:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.64, 3.95 [02:07:52] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [02:08:29] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:29] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.249 second response time [02:08:51] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [02:09:01] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.009 second response time [02:09:26] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:10:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.72, 1.96 [02:13:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.29, 3.60 [02:14:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.59, 3.54 [02:15:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 10.58, 9.98 [02:17:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.75, 9.87, 9.78 [02:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.61, 1.93, 1.92 [02:18:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.53, 3.42, 3.51 [02:19:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.64, 3.32, 3.66 [02:20:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.41, 3.48 [02:22:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.87, 3.12, 3.36 [02:24:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.92, 1.96 [02:25:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 2.90, 3.36 [02:28:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.64, 3.46 [02:30:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.35, 3.36 [02:32:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.54, 3.53 [02:32:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 3.05, 3.25 [02:33:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.19, 9.39, 9.27 [02:34:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.44, 4.08, 3.72 [02:35:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.16, 9.14, 9.21 [02:37:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.62, 1.74, 1.35 [02:38:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.57, 3.62 [02:39:35] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.23, 1.50, 1.31 [02:40:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.58, 1.58, 1.70 [02:42:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.51, 3.75, 3.62 [02:46:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.68, 3.65 [02:50:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.11, 2.52, 2.03 [02:52:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.35, 3.04, 3.36 [02:58:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.87, 1.93 [03:12:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.80, 1.80 [03:25:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.87, 1.47 [03:27:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 4.37, 3.49 [03:29:13] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.17, 1.52, 1.42 [03:31:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.63, 3.41 [03:33:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.18, 3.26 [03:36:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.94, 1.99 [03:37:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.46, 4.25, 3.67 [03:38:35] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [03:38:37] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.071 second response time [03:39:37] PROBLEM - db112 MariaDB on db112 is CRITICAL: Too many connections [03:41:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.67, 3.58 [03:43:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.71, 3.59 [03:47:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.94, 3.76 [03:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.61, 2.09, 1.98 [03:51:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.14, 4.28, 3.86 [03:53:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.16, 5.25, 3.87 [03:54:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.52, 3.19 [03:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.96, 1.98 [04:02:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.76, 10.27, 8.58 [04:03:47] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.32, 2.38, 1.75 [04:04:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.46, 9.02, 8.35 [04:05:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.41, 2.00, 1.69 [04:09:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.46, 1.54 [04:10:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.48, 1.56, 1.69 [04:14:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 10.92, 9.20 [04:15:53] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:16:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:16:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.09, 11.24, 9.13 [04:17:53] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.333 second response time [04:18:05] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23900 bytes in 0.330 second response time [04:18:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.24, 10.17, 8.99 [04:18:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.63, 10.66, 9.58 [04:20:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.46, 2.11, 1.88 [04:20:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.05, 9.96, 9.43 [04:21:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.56, 3.87 [04:24:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.85, 1.84 [04:27:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.62, 3.74 [04:30:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.53, 1.70 [04:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.76, 1.76 [04:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.84, 1.77 [04:49:09] !log running delbackups (bash ./delbackups.sh /home/reception/delwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php) [04:49:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.26, 3.89 [04:57:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.22, 4.76, 4.29 [05:01:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.15, 3.54, 3.91 [05:01:41] !log manually run matomo script (sudo -u www-data /usr/local/bin/runMatomoArchive.sh) [05:01:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:01] hmm, returned empty [05:04:31] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 99069 bytes in 2.106 second response time [05:04:35] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.133 second response time [05:04:37] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.169 second response time [05:04:37] !log removed last 3 binlogs on db112 to clear space [05:04:47] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 1.341 second response time [05:04:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:05:02] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 99077 bytes in 1.052 second response time [05:05:37] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 120605 Threads: 38 Questions: 53871554 Slow queries: 680534 Opens: 10330 Open tables: 1357 Queries per second avg: 446.677 [05:05:42] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.590 second response time [05:06:06] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.650 second response time [05:06:24] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6153 bytes in 0.107 second response time [05:06:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.65, 3.95 [05:08:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.93, 2.00 [05:12:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 10.33, 9.52 [05:14:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 2.03, 2.02 [05:14:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.78, 3.78 [05:14:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.02, 9.47, 9.31 [05:15:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=dreamscapewiki --report 1 --username-prefix=w /home/reception/Wikipedia-20220608024818.xml (START) [05:15:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:15:31] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=dreamscapewiki --report 1 --username-prefix=w /home/reception/Wikipedia-20220608024818.xml (END - exit=0) [05:15:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.60, 3.71 [05:17:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.69, 3.30 [05:18:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.62, 3.69 [05:18:53] !log RENAMED evilgeniuswikiwiki to evilgeniuswiki [05:19:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:21:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.10, 10.27, 9.08 [05:21:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.30, 6.33, 5.50 [05:22:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 10.71, 9.80 [05:22:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.74, 3.72 [05:23:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.79, 9.43, 8.90 [05:23:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.86, 6.19, 5.55 [05:24:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.15, 9.32, 9.40 [05:24:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 3.93, 3.78 [05:28:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.52, 3.46 [05:28:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.97, 3.85 [05:30:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.18, 3.34 [05:30:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.92, 3.84 [05:32:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.75, 1.96 [05:32:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.16, 3.33, 3.64 [05:33:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.41, 3.91 [05:38:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.66, 3.63 [05:39:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 4.30, 4.10 [05:40:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 9.98, 9.47 [05:40:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.47, 3.56 [05:41:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.45, 3.32 [05:42:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.71, 8.77, 9.09 [05:42:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 4.10, 3.81 [05:42:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.88, 3.51 [05:44:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.73, 3.70 [05:44:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.42, 3.39 [05:46:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.14, 3.29 [05:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 1.94, 1.84 [05:52:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 4.03, 3.78 [05:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.89, 1.88 [05:54:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.75, 3.71 [06:02:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.04, 3.01, 3.39 [06:04:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.77, 3.41 [06:06:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.85, 1.81 [06:06:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 3.10, 3.20 [06:07:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.33, 3.85 [06:13:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.96, 4.35, 4.10 [06:14:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.16, 3.26, 1.83 [06:18:14] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.45, 1.72, 1.51 [06:19:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.50, 3.79, 3.38 [06:20:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.37, 3.09 [06:20:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.76, 1.39, 1.41 [06:20:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.63, 10.10, 8.83 [06:21:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.85, 3.46 [06:22:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 3.19, 3.05 [06:22:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.21, 10.00, 9.28 [06:24:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 9.29, 9.12 [06:24:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 8.84, 8.62 [06:24:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.67, 3.05, 3.25 [06:26:10] !log deleted further binlogs from 12 June [06:26:15] !log (on db112) [06:26:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 3.78, 3.26 [06:28:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 4.26, 3.72 [06:29:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.35, 3.16 [06:29:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 10.85, 9.75 [06:30:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.90, 10.64, 9.36 [06:30:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.77, 3.60 [06:31:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 10.57, 9.81 [06:32:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.72, 10.69, 9.54 [06:35:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.57, 3.90 [06:35:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 11.51, 10.31 [06:35:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.13, 9.87, 9.46 [06:38:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.90, 2.85, 3.27 [06:39:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.88, 11.12, 10.05 [06:40:27] before deletion: [06:40:38] db101: 274G available [06:40:46] db111: 305G available [06:40:59] db121: 255G availabler [06:41:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.75, 11.04, 10.51 [06:43:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.87, 9.05, 9.49 [06:43:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.14, 9.71, 10.08 [06:47:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.14, 3.13, 3.40 [06:49:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.01, 10.34, 9.88 [06:49:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.69, 10.91, 10.42 [06:50:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.42, 3.07 [06:51:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.99, 9.21, 9.53 [06:51:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.55, 3.23 [06:51:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 4.14, 3.76 [06:52:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 3.24, 3.06 [06:53:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.28, 3.03, 3.07 [06:53:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 9.80, 10.12 [06:53:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.37, 3.52 [06:57:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.77, 3.25 [07:03:08] !log running deletedWikis.php [07:03:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:32] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.22, 7.06, 5.96 [07:12:23] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.33, 6.46, 5.99 [07:15:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.50, 3.09 [07:15:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.33, 2.89 [07:16:32] gluster101: 50G available (before deletions) [07:17:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.89, 2.78 [07:17:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.36, 3.09 [07:21:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.36, 3.51 [07:22:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.37, 4.33, 3.33 [07:23:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.18, 3.47, 3.29 [07:23:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.60, 3.18 [07:25:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.93, 3.25, 3.09 [07:28:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 9.73, 9.10 [07:30:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.09, 8.76, 8.83 [07:31:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.19, 3.26 [07:35:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 3.99, 3.42 [07:35:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.83, 5.01, 4.03 [07:36:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.90, 3.39 [07:38:35] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:38:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 10.55, 9.48 [07:40:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 9.27, 9.14 [07:41:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.94, 3.60 [07:44:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.48, 3.45 [07:46:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 3.14, 3.32 [07:46:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.80, 3.06, 3.29 [07:47:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.74, 3.90 [07:49:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.31, 9.82, 8.78 [07:51:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.04, 8.89, 8.55 [07:51:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 3.57, 3.73 [07:57:27] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.87, 1.67, 1.29 [07:57:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.77, 3.44 [07:57:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.59, 3.70 [07:59:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.50, 3.25, 3.29 [07:59:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 4.00, 3.85 [08:01:23] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.70, 1.62, 1.35 [08:03:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.79, 3.30, 3.64 [08:05:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.21, 4.81, 4.19 [08:08:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:09:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.95, 3.99 [08:11:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 4.23, 4.08 [08:13:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.91, 3.98 [08:23:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 10.37, 9.30 [08:25:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 2.66, 3.22 [08:27:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.88, 9.60, 9.25 [08:34:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:34:59] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:36:48] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.329 second response time [08:36:56] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.017 second response time [08:37:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 3.44, 2.98 [08:38:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.27, 4.28, 3.66 [08:39:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.29, 2.85 [08:39:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.36, 3.00 [08:40:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.97, 3.61 [08:41:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.90, 2.77 [08:43:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.19, 3.24, 3.03 [08:44:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.66, 3.53 [08:46:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.84, 3.18, 3.37 [08:57:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.36, 1.95, 1.48 [09:01:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.98, 2.45, 1.72 [09:05:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.90, 1.63, 1.55 [09:06:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.56, 3.39 [09:11:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.42, 3.37 [09:13:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.82, 3.53 [09:15:42] [02ssl] 07xrqulette opened pull request 03#548: Create tr.wikiretet.ga.crt - 13https://github.com/miraheze/ssl/pull/548 [09:15:42] [url] Page not found · GitHub · GitHub | github.com [09:15:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.80, 3.58 [09:17:55] [02dns] 07xrqulette opened pull request 03#289: Pull request - 13https://github.com/miraheze/dns/pull/289 [09:17:55] [url] Page not found · GitHub · GitHub | github.com [09:19:46] [02ssl] 07Universal-Omega commented on pull request 03#548: Create tr.wikiretet.ga.crt - 13https://github.com/miraheze/ssl/pull/548#issuecomment-1154932685 [09:19:47] [url] Create tr.wikiretet.ga.crt by xrqulette · Pull Request #548 · miraheze/ssl · GitHub | github.com [09:19:48] [02ssl] 07Universal-Omega closed pull request 03#548: Create tr.wikiretet.ga.crt - 13https://github.com/miraheze/ssl/pull/548 [09:19:49] [url] Page not found · GitHub · GitHub | github.com [09:21:00] [02ssl] 07Universal-Omega edited a comment on pull request 03#548: Create tr.wikiretet.ga.crt - 13https://github.com/miraheze/ssl/pull/548#issuecomment-1154932685 [09:21:01] [url] Create tr.wikiretet.ga.crt by xrqulette · Pull Request #548 · miraheze/ssl · GitHub | github.com [09:21:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.76, 3.61 [09:22:21] [02ssl] 07Universal-Omega edited a comment on pull request 03#548: Create tr.wikiretet.ga.crt - 13https://github.com/miraheze/ssl/pull/548#issuecomment-1154932685 [09:22:21] [url] Create tr.wikiretet.ga.crt by xrqulette · Pull Request #548 · miraheze/ssl · GitHub | github.com [09:23:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.31, 9.50 [09:23:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 3.10, 3.38 [09:25:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.91, 9.34, 9.24 [09:30:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.20, 2.92 [09:31:45] Reception123: Run deletewikis script! [09:31:48] Reception123: Run deletewikis script! [09:32:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.87, 2.65, 2.75 [09:49:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.73, 3.46 [09:51:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.94, 3.58 [09:53:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.87, 3.61 [09:55:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.27, 1.14 [09:56:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.71, 3.20 [09:57:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.58, 1.29, 1.16 [09:59:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.36, 4.84, 4.01 [09:59:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 4.21, 3.27 [10:00:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.71, 1.96 [10:01:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.70, 3.20 [10:02:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 3.37, 3.37 [10:02:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.61, 3.70 [10:03:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.70, 2.93, 2.97 [10:04:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.69, 4.20, 3.89 [10:06:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.54, 3.67 [10:08:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.90, 1.93 [10:08:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.48, 4.68, 4.04 [10:09:00] [02mediawiki] 07Universal-Omega synchronize pull request 03#7621: Automatically merge i18n and build only - 13https://github.com/miraheze/mediawiki/pull/7621 [10:09:01] [url] Page not found · GitHub · GitHub | github.com [10:09:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/4b6061bcc800...acc9206389a9 [10:09:04] [url] Comparing 4b6061bcc800...acc9206389a9 · miraheze/mediawiki · GitHub | github.com [10:09:04] [02miraheze/mediawiki] 07Universal-Omega 03acc9206 - Update automerge.yml [10:09:14] [02mediawiki] 07Universal-Omega edited pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [10:09:14] [url] Page not found · GitHub · GitHub | github.com [10:09:46] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/acc9206389a9...cc42cf01d6b9 [10:09:47] [url] Comparing acc9206389a9...cc42cf01d6b9 · miraheze/mediawiki · GitHub | github.com [10:09:47] [02miraheze/mediawiki] 07Universal-Omega 03cc42cf0 - Update automerge.yml [10:09:49] [02mediawiki] 07Universal-Omega synchronize pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [10:09:49] [url] Page not found · GitHub · GitHub | github.com [10:12:33] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.88, 3.84 [10:12:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/QuickInstantCommons-b07cab1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5b5c303fe4b7 [10:12:52] [02miraheze/mediawiki] 07dependabot[bot] 035b5c303 - Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` [10:12:53] [02mediawiki] 07dependabot[bot] opened pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [10:12:54] [url] Page not found · GitHub · GitHub | github.com [10:12:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/QuickInstantCommons-b07cab1 - 13https://github.com/miraheze/mediawiki [10:12:55] [url] Page not found · GitHub · GitHub | github.com [10:12:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/cc42cf01d6b9...886d578fd791 [10:12:57] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:57] [url] Comparing cc42cf01d6b9...886d578fd791 · miraheze/mediawiki · GitHub | github.com [10:12:58] [02miraheze/mediawiki] 07Universal-Omega 03886d578 - Update automerge.yml [10:12:59] [02mediawiki] 07Universal-Omega synchronize pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [10:13:00] ... [10:13:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SocialProfile-280eb97 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/dd0d122391cb [10:13:03] [02miraheze/mediawiki] 07dependabot[bot] 03dd0d122 - Bump extensions/SocialProfile from `932e4d9` to `280eb97` [10:13:04] [02mediawiki] 07dependabot[bot] opened pull request 03#7623: Bump extensions/SocialProfile from `932e4d9` to `280eb97` - 13https://github.com/miraheze/mediawiki/pull/7623 [10:13:05] [url] Page not found · GitHub · GitHub | github.com [10:13:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-280eb97 - 13https://github.com/miraheze/mediawiki [10:13:06] ... [10:13:06] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageForms-7847a4d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e0153fddb377 [10:13:09] [02miraheze/mediawiki] 07dependabot[bot] 03e0153fd - Bump extensions/PageForms from `0440bbc` to `7847a4d` [10:13:10] [02mediawiki] 07dependabot[bot] opened pull request 03#7624: Bump extensions/PageForms from `0440bbc` to `7847a4d` - 13https://github.com/miraheze/mediawiki/pull/7624 [10:13:10] [url] Page not found · GitHub · GitHub | github.com [10:13:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageForms-7847a4d - 13https://github.com/miraheze/mediawiki [10:13:12] ... [10:13:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-f182386 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3998f3c4c66c [10:13:14] [02miraheze/mediawiki] 07dependabot[bot] 033998f3c - Bump extensions/Translate from `3b07a4b` to `f182386` [10:13:16] [02mediawiki] 07dependabot[bot] opened pull request 03#7625: Bump extensions/Translate from `3b07a4b` to `f182386` - 13https://github.com/miraheze/mediawiki/pull/7625 [10:13:17] [url] Page not found · GitHub · GitHub | github.com [10:13:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-f182386 - 13https://github.com/miraheze/mediawiki [10:13:18] [url] Page not found · GitHub · GitHub | github.com [10:13:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [10:13:20] [url] Page not found · GitHub · GitHub | github.com [10:13:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.44, 10.01, 8.94 [10:13:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cargo-751b633 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/98dc9d07854b [10:13:26] [02miraheze/mediawiki] 07dependabot[bot] 0398dc9d0 - Bump extensions/Cargo from `ed30677` to `751b633` [10:13:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cargo-751b633 - 13https://github.com/miraheze/mediawiki [10:13:29] [02mediawiki] 07dependabot[bot] opened pull request 03#7626: Bump extensions/Cargo from `ed30677` to `751b633` - 13https://github.com/miraheze/mediawiki/pull/7626 [10:13:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#7624: Bump extensions/PageForms from `0440bbc` to `7847a4d` - 13https://github.com/miraheze/mediawiki/pull/7624 [10:13:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#7624: Bump extensions/PageForms from `0440bbc` to `7847a4d` - 13https://github.com/miraheze/mediawiki/pull/7624 [10:13:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#7623: Bump extensions/SocialProfile from `932e4d9` to `280eb97` - 13https://github.com/miraheze/mediawiki/pull/7623 [10:13:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.35, 9.94 [10:13:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#7625: Bump extensions/Translate from `3b07a4b` to `f182386` - 13https://github.com/miraheze/mediawiki/pull/7625 [10:13:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#7625: Bump extensions/Translate from `3b07a4b` to `f182386` - 13https://github.com/miraheze/mediawiki/pull/7625 [10:13:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#7626: Bump extensions/Cargo from `ed30677` to `751b633` - 13https://github.com/miraheze/mediawiki/pull/7626 [10:14:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.312 second response time [10:14:54] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.094 second response time [10:15:06] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.519 second response time [10:15:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.71, 8.72, 8.59 [10:15:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.46, 3.25 [10:17:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.29, 3.22 [10:17:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.27, 8.00, 8.94 [10:18:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.10, 4.80, 4.21 [10:20:16] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.53, 3.38 [10:21:16] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:17] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.35, 10.27, 9.56 [10:21:37] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:42] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:02] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:09] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:11] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.481 second response time [10:23:13] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.319 second response time [10:23:14] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [10:23:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.07, 10.43, 9.19 [10:23:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.41, 10.55, 9.77 [10:23:33] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [10:23:36] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.269 second response time [10:23:37] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.307 second response time [10:24:00] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.776 second response time [10:24:08] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.022 second response time [10:24:18] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23900 bytes in 0.497 second response time [10:24:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.74, 3.96 [10:25:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.76, 9.94, 9.18 [10:25:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.84, 9.55, 9.50 [10:26:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.20, 3.30 [10:32:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.81, 3.86 [10:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.80, 1.96 [10:36:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.40, 4.53, 3.80 [10:38:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.90, 3.66 [10:39:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.53, 3.38 [10:40:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.73, 3.95 [10:41:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.84, 2.96, 3.18 [10:44:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.78, 3.88 [10:46:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.89, 3.91 [10:47:17] PROBLEM - otogebase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'otogebase.com' expires in 15 day(s) (Thu 30 Jun 2022 10:27:01 GMT +0000). [10:47:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.72, 0.95 [10:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.54, 1.95, 1.89 [10:48:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.32, 3.14, 3.31 [10:49:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.38, 0.91 [10:53:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9ae71bea8b80...7a42acb99bf1 [10:53:44] [url] Comparing 9ae71bea8b80...7a42acb99bf1 · miraheze/ssl · GitHub | github.com [10:53:44] [02miraheze/ssl] 07MirahezeSSLBot 037a42acb - Bot: Update SSL cert for otogebase.com [10:53:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.27, 4.90, 4.18 [10:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.97, 1.94 [10:59:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 4.04, 3.56 [11:01:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.75, 3.51 [11:03:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.85, 9.60, 8.68 [11:04:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 10.11, 9.10 [11:05:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.54, 9.11, 8.61 [11:05:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.20, 2.98, 3.25 [11:06:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.30, 9.59, 9.05 [11:09:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.36, 3.87 [11:10:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.96, 11.49, 9.90 [11:11:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 3.76, 3.94 [11:12:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 11.25, 10.02 [11:13:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.39, 3.79 [11:14:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.48, 1.58, 1.70 [11:15:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.79, 3.89 [11:16:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.53, 10.09, 9.79 [11:17:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.54, 3.79 [11:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.43, 1.98, 1.83 [11:23:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.66, 10.52, 10.05 [11:23:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 3.76, 3.69 [11:26:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.93, 1.93 [11:27:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.07, 9.38, 9.72 [11:27:58] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.14, 1.58, 1.19 [11:28:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.36, 3.66, 3.08 [11:29:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 4.25, 3.43 [11:29:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.41, 1.18 [11:31:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.64, 10.13 [11:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.91, 1.91 [11:32:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.21, 9.77, 9.37 [11:33:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 10.29, 10.09 [11:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.79, 1.87 [11:34:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.93, 8.99, 9.12 [11:35:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 10.80, 10.27 [11:38:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.09, 11.10, 9.93 [11:40:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.94, 10.56, 9.87 [11:41:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.09, 10.54, 10.52 [11:43:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.93, 3.99 [11:44:06] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [11:45:17] RECOVERY - otogebase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'otogebase.com' will expire on Mon 12 Sep 2022 09:53:37 GMT +0000. [11:46:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.52, 11.53, 10.43 [11:46:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 10.94, 10.62 [11:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.79, 2.07, 1.91 [11:48:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.04, 10.27, 10.40 [11:49:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.90, 11.66, 10.74 [11:53:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.35, 1.65, 1.32 [11:54:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.45, 3.90 [11:55:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 2.97, 3.39 [11:55:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.07, 1.47, 1.30 [11:57:58] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.52, 1.21 [11:59:06] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [11:59:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.56, 3.54 [11:59:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.37, 4.00 [11:59:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.85, 1.24, 1.15 [12:01:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.39, 8.37, 9.79 [12:01:52] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:57] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:12] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:23] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.20, 9.22, 10.00 [12:02:59] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:03] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:16] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.45, 3.50 [12:03:28] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:39] !log sudo -u www-data /usr/local/bin/runMatomoArchive.sh (this time with DB working) [12:03:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.66, 3.96 [12:03:53] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 3.526 second response time [12:03:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:03:58] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 3.167 second response time [12:04:04] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.517 second response time [12:04:11] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 3.379 second response time [12:04:24] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.317 second response time [12:04:58] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.224 second response time [12:04:59] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.397 second response time [12:05:11] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.309 second response time [12:05:27] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23900 bytes in 1.184 second response time [12:05:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.77, 3.98 [12:06:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 10.59, 10.34 [12:06:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 4.05, 3.80 [12:07:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.68, 3.57 [12:07:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.13, 4.59, 4.25 [12:09:01] !log deleted 3 binlogs on db112 from June 12 [12:09:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:10:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.35, 9.83, 10.18 [12:11:11] paladox: what do you think about https://phabricator.miraheze.org/T9383#190033 ? [12:11:11] [url] ⚓ T9383 db112 disk space very low | phabricator.miraheze.org [12:11:27] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:33] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:54] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:34] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:37] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:22] Reception123: sounds fine with me if you want to do that [12:13:51] paladox: I'd wait for John before, but was just wondering if it'd be possible [12:14:10] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.98, 7.01, 5.95 [12:14:32] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 1.139 second response time [12:14:35] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.753 second response time [12:15:04] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:31] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.758 second response time [12:15:42] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.324 second response time [12:15:54] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.306 second response time [12:16:06] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.04, 5.88, 5.66 [12:16:57] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.341 second response time [12:17:04] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 914 bytes in 0.234 second response time [12:17:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:38] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:42] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.70, 11.35, 10.45 [12:18:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [12:18:56] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 4.349 second response time [12:19:10] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23900 bytes in 4.453 second response time [12:19:37] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.072 second response time [12:19:38] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.067 second response time [12:20:26] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [12:21:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.46, 11.41, 10.71 [12:21:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.60, 7.29, 6.30 [12:23:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.60, 6.80, 6.26 [12:25:49] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:10] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 11.08, 10.30 [12:27:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.32, 3.90 [12:27:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.24, 11.65, 10.97 [12:27:39] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:11] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 8.822 second response time [12:28:11] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 7.023 second response time [12:28:25] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 5.595 second response time [12:28:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 11.33, 10.50 [12:29:06] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:29:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.79, 4.21, 4.13 [12:29:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 10.90, 10.78 [12:29:39] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.348 second response time [12:29:56] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.516 second response time [12:32:45] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.85, 7.49, 6.71 [12:33:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.57, 11.02, 10.80 [12:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.64, 1.96 [12:34:40] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.14, 7.84, 6.91 [12:35:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.90, 11.23, 10.91 [12:36:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.65, 11.82, 10.87 [12:36:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.31, 7.45, 6.87 [12:37:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.67, 3.96 [12:37:58] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.56, 1.18 [12:38:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 1.84, 1.95 [12:38:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 10.90, 10.66 [12:38:32] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.53, 7.67, 7.02 [12:39:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.02, 11.46, 11.04 [12:39:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 4.16, 4.12 [12:39:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.82, 1.28, 1.12 [12:40:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.71, 1.89 [12:40:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.59, 7.10, 6.88 [12:41:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.25, 10.19, 10.66 [12:41:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.82, 3.59, 3.91 [12:42:23] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.36, 6.59, 6.73 [12:43:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.88, 3.97 [12:44:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.58, 4.00 [12:45:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.33, 10.47, 10.56 [12:45:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.71, 3.88 [12:46:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 4.13, 4.14 [12:47:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.69, 10.75, 10.65 [12:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 1.89, 1.86 [12:48:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:32] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 2.606 second response time [12:50:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 9.64, 8.65 [12:51:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.27, 9.59, 8.63 [12:52:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.67, 3.95 [12:52:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 8.72, 8.44 [12:53:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.64, 9.26, 10.10 [12:53:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.15, 8.60, 8.39 [12:54:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 10.78, 10.62 [12:55:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:58] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.014 second response time [12:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.90, 1.94 [12:56:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:47] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.47, 3.78 [12:56:55] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.87, 10.26, 10.31 [12:57:44] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 7.234 second response time [12:57:58] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.566 second response time [12:58:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.21, 11.75, 11.11 [12:58:41] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 1.304 second response time [12:58:46] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.320 second response time [12:58:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.366 second response time [12:59:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.75, 9.62, 10.07 [12:59:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.97, 4.22, 3.92 [13:00:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.62, 3.78 [13:01:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.99, 3.88 [13:04:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.96, 11.87, 11.27 [13:06:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.14, 10.40, 10.83 [13:08:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.10, 3.98, 3.79 [13:10:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.55, 1.69 [13:10:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.31, 3.38, 3.60 [13:11:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.87, 3.37 [13:11:34] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:05] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:13] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:10] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:20] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:30] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.347 second response time [13:14:10] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.611 second response time [13:15:05] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:06] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:07] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:11] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:24] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:54] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:11] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [13:16:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.05, 9.67, 10.19 [13:17:24] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 6.676 second response time [13:17:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 2.925 second response time [13:17:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.017 second response time [13:18:11] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [13:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 2.04, 1.84 [13:18:24] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.396 second response time [13:19:04] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.336 second response time [13:19:06] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.335 second response time [13:19:07] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.677 second response time [13:19:09] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.019 second response time [13:19:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.39, 10.90, 10.14 [13:19:20] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.281 second response time [13:19:31] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.993 second response time [13:21:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.41, 10.52, 10.09 [13:21:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.18, 8.76, 7.59 [13:21:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.63, 3.17, 3.97 [13:22:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.75, 3.66 [13:23:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.79, 10.28, 10.22 [13:23:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.91, 7.92, 7.42 [13:23:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.90, 3.82, 4.13 [13:24:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.67, 1.82, 1.18 [13:25:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.60, 9.66, 9.90 [13:26:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.45, 1.64, 1.19 [13:27:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.46, 4.47, 3.73 [13:27:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.90, 9.78, 10.06 [13:28:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.94, 1.97 [13:32:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.91, 3.92 [13:33:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 10.80, 10.31 [13:33:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.93, 11.00, 10.38 [13:34:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.22, 4.71, 4.20 [13:35:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.46, 9.65, 9.96 [13:37:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 11.02, 10.56 [13:38:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.83, 1.85 [13:39:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 11.48, 10.77 [13:39:06] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:39:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.40, 11.02, 10.38 [13:40:51] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:48] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.018 second response time [13:43:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 11.74, 11.08 [13:43:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 10.35, 10.36 [13:46:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.53, 3.99 [13:48:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.60, 3.93 [13:49:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 2.10, 1.39 [13:50:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.47, 3.84 [13:51:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.68, 1.33 [13:52:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.25, 3.84 [13:53:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.57, 10.00, 10.17 [13:57:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.57, 6.87, 6.04 [13:58:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.65, 3.77 [14:00:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.74, 10.26, 9.06 [14:00:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.30, 3.63 [14:01:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.75, 6.43, 6.10 [14:02:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.78, 9.60, 8.98 [14:02:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.68, 3.67 [14:04:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.85, 3.06, 3.45 [14:06:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.42, 11.19, 9.70 [14:06:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 2.75, 3.29 [14:06:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:06:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.79, 9.49, 10.15 [14:07:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.83, 10.11, 9.01 [14:08:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.63, 11.29, 9.92 [14:08:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.87, 2.00 [14:08:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.55, 3.55 [14:09:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.66, 9.59, 8.95 [14:09:57] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:10:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.56, 9.75, 9.54 [14:10:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.37, 3.48 [14:12:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 3.14, 3.38 [14:16:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.70, 3.54 [14:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.00, 2.03, 1.94 [14:18:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 3.13, 3.34 [14:22:45] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.61, 3.06 [14:22:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.54, 3.44 [14:22:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.83, 10.35, 9.44 [14:22:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.56, 3.32 [14:24:40] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.12, 3.09, 2.94 [14:24:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.54, 3.45 [14:24:52] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.50, 10.07, 9.46 [14:26:00] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.89, 2.00, 1.54 [14:26:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:26:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.85, 3.58 [14:30:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.87, 2.00 [14:32:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.21, 11.00, 9.90 [14:32:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.70, 3.62 [14:32:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.90, 3.79 [14:33:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 11.08, 10.19 [14:34:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.56, 9.96, 9.65 [14:34:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.77, 3.65 [14:34:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.93, 4.60, 4.05 [14:35:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.74, 11.10, 10.30 [14:35:58] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.87, 1.87, 1.84 [14:37:42] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:54] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.77, 3.74 [14:39:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.99, 11.26, 10.47 [14:39:30] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [14:39:31] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:39:50] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:12] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.97, 11.18, 10.54 [14:41:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.00, 1.29, 1.60 [14:41:59] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 9.357 second response time [14:42:04] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 5.708 second response time [14:42:13] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.583 second response time [14:43:33] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 1.389 second response time [14:43:45] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.769 second response time [14:44:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.06, 2.05, 1.63 [14:45:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.85, 10.42, 9.36 [14:46:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 4.23, 3.95 [14:46:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.56, 3.98 [14:47:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.46, 9.76, 10.12 [14:47:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.29, 10.02, 9.33 [14:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 1.88, 1.82 [14:48:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.69, 3.79 [14:49:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 8.85, 8.98 [14:49:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.02, 1.81, 1.71 [14:49:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.50, 12.11, 10.44 [14:50:11] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.59, 3.85 [14:50:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 11.69, 10.88 [14:51:39] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.69, 4.21, 3.27 [14:51:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.18, 1.61, 1.65 [14:52:06] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.329 second response time [14:52:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 4.01, 3.87 [14:53:35] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.51, 3.59, 3.16 [14:54:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.31, 12.15, 11.17 [14:55:30] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.20, 3.11, 3.03 [14:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.87, 1.90 [14:56:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.65, 3.76 [14:56:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.65, 3.83 [14:57:39] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:59:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.98, 11.87, 11.09 [15:00:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 1.96, 1.93 [15:00:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.51, 10.24, 10.83 [15:01:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.11, 10.78, 9.39 [15:02:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 2.82, 3.38 [15:03:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.77, 10.15, 9.35 [15:04:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.93, 1.92 [15:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.91, 11.59, 11.20 [15:04:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f33b4ddde58a [15:04:57] [02miraheze/puppet] 07paladox 03f33b4dd - matomo: set bulk_requests_use_transaction to 0 [15:04:58] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [15:04:59] [url] Page not found · GitHub · GitHub | github.com [15:05:00] [02puppet] 07paladox opened pull request 03#2670: matomo: set bulk_requests_use_transaction to 0 - 13https://github.com/miraheze/puppet/pull/2670 [15:05:00] ... [15:05:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.19, 11.71, 11.20 [15:05:55] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 8.77, 7.28, 3.90 [15:06:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 2.87, 3.31 [15:07:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.07, 11.67, 11.27 [15:08:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.93, 1.92 [15:09:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.67, 3.56 [15:10:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.84, 1.89 [15:10:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 11.10, 11.20 [15:12:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 9.38, 7.97 [15:13:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.58, 9.93, 9.03 [15:13:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.09, 3.35 [15:14:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.45, 8.41, 7.80 [15:15:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.47, 9.00, 8.80 [15:16:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.80, 1.84 [15:17:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.16, 10.78, 10.79 [15:17:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.40, 3.91 [15:18:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.62, 11.25 [15:19:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.10, 11.06, 10.91 [15:20:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 10.81, 11.00 [15:22:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.33, 3.26 [15:23:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.60, 4.80, 4.25 [15:24:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.79, 3.44 [15:26:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.71, 3.36 [15:27:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.07, 1.36, 1.96 [15:30:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.02, 8.38, 9.98 [15:30:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.70, 3.49 [15:31:11] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.34, 8.55, 9.88 [15:31:21] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:09] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 2.004 second response time [15:33:21] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.406 second response time [15:33:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.56, 1.02, 1.61 [15:34:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.73, 4.08, 3.71 [15:36:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.61, 10.32, 10.20 [15:38:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 10.47, 10.27 [15:40:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.29, 9.42, 8.16 [15:40:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.15, 9.58, 9.97 [15:42:10] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.530 second response time [15:42:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.56, 9.21, 8.25 [15:46:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.19, 10.12, 9.30 [15:46:55] !log [rhinos@test101] starting deploy of {'world': True, 'l10n': True, 'force': True, 'ignoretime': True} to test101 [15:47:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:47:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.33, 10.11, 9.48 [15:48:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.48, 9.47, 9.17 [15:48:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 10.40, 10.12 [15:49:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.79, 10.53, 9.72 [15:50:31] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/2eb706db0455 [15:50:32] [02miraheze/ssl] 07MacFan4000 032eb706d - add tr.wikiretet.ga [15:50:34] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [15:50:34] [url] Page not found · GitHub · GitHub | github.com [15:51:32] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:42] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.35, 9.09, 9.30 [15:51:46] [02ssl] 07MacFan4000 opened pull request 03#549: add tr.wikiretet.ga - 13https://github.com/miraheze/ssl/pull/549 [15:51:47] ... [15:52:11] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.59, 12.32, 10.45 [15:52:25] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.60, 9.86, 9.99 [15:53:07] !log [rhinos@mwtask111] finished deploy of {'world': True, 'l10n': True, 'force': True, 'servers': 'mw101', 'ignoretime': True} to mw101 - SUCCESS in 290s [15:53:09] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:14] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2eb706db0455...021a8c015c89 [15:54:15] [url] Comparing 2eb706db0455...021a8c015c89 · miraheze/ssl · GitHub | github.com [15:54:16] [02miraheze/ssl] 07MacFan4000 03021a8c0 - Update certs.yaml [15:54:17] [02ssl] 07MacFan4000 synchronize pull request 03#549: add tr.wikiretet.ga - 13https://github.com/miraheze/ssl/pull/549 [15:54:18] [url] Page not found · GitHub · GitHub | github.com [15:54:49] [02ssl] 07MacFan4000 closed pull request 03#549: add tr.wikiretet.ga - 13https://github.com/miraheze/ssl/pull/549 [15:54:49] [url] Page not found · GitHub · GitHub | github.com [15:54:50] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [15:54:52] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/7a42acb99bf1...65ef188ddb8b [15:54:52] [url] Comparing 7a42acb99bf1...65ef188ddb8b · miraheze/ssl · GitHub | github.com [15:54:53] [02miraheze/ssl] 07MacFan4000 0365ef188 - add tr.wikiretet.ga (#549) [15:54:55] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [15:54:55] [url] Page not found · GitHub · GitHub | github.com [15:57:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/07eaadf42037...fbca8a19badb [15:57:26] [url] Comparing 07eaadf42037...fbca8a19badb · miraheze/mw-config · GitHub | github.com [15:57:27] [02miraheze/mw-config] 07Universal-Omega 03fbca8a1 - Fix [15:57:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.38, 1.74, 1.39 [15:58:40] miraheze/mw-config - Universal-Omega the build passed. [15:58:53] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 9.347 second response time [15:59:12] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 2.415 second response time [15:59:35] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.65, 1.41 [15:59:42] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.020 second response time [15:59:45] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.328 second response time [16:00:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.85, 11.16, 10.82 [16:00:40] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.316 second response time [16:00:44] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.72, 9.75, 9.26 [16:02:23] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:02:45] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.257 second response time [16:05:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 10.54, 10.08 [16:05:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 3.72, 2.21 [16:06:56] [02puppet] 07RhinosF1 opened pull request 03#2671: upgradetool: switch to new mass plan - 13https://github.com/miraheze/puppet/pull/2671 [16:06:57] [url] Page not found · GitHub · GitHub | github.com [16:08:10] [02puppet] 07RhinosF1 synchronize pull request 03#2671: upgradetool: switch to new mass plan - 13https://github.com/miraheze/puppet/pull/2671 [16:08:10] [url] Page not found · GitHub · GitHub | github.com [16:11:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.83, 1.80 [16:12:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.39, 10.72, 10.58 [16:13:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 11.80, 10.84 [16:13:51] !log [rhinos@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [16:14:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.65, 10.75, 10.62 [16:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:05] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.97, 6.87, 5.97 [16:15:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.82, 11.91, 11.00 [16:15:23] RECOVERY - test101 APT on test101 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:15:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.29, 1.59 [16:17:00] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.97, 6.57, 5.97 [16:17:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.90, 12.18, 11.23 [16:17:13] !log ignore deploy error [16:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.24, 10.92, 10.90 [16:20:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.35, 11.05, 10.76 [16:20:13] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [16:21:06] !log [@test101] starting deploy of {'config': True} to all [16:21:09] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 2s [16:21:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:21:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.81, 10.47, 9.48 [16:21:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.43, 11.50, 10.99 [16:22:13] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [16:23:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.27, 9.35, 9.20 [16:23:38] !log [@mwtask111] starting deploy of {'config': True} to all [16:23:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:16] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 38s [16:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:07] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:09] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:11] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:28] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 2.00, 1.77, 1.57 [16:26:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.00, 2.03, 1.78 [16:27:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.38, 9.30, 10.13 [16:31:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 11.59, 10.99 [16:31:33] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.90, 1.44, 1.51 [16:31:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.54, 10.39, 10.34 [16:31:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.47, 1.22, 1.50 [16:33:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 11.47, 11.02 [16:33:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 7.294 second response time [16:33:33] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.792 second response time [16:33:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.20, 10.18, 10.27 [16:33:40] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.232 second response time [16:33:46] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 1.841 second response time [16:37:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.42, 9.06, 9.86 [16:37:26] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.34, 1.75, 1.63 [16:37:44] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:55] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:04] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:07] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:11] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:18] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:13] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.41, 1.29, 1.48 [16:39:36] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:08] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.387 second response time [16:40:22] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 8.453 second response time [16:41:09] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.324 second response time [16:41:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.58, 4.00 [16:41:35] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.021 second response time [16:41:43] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.386 second response time [16:41:51] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 5.027 second response time [16:41:52] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 5.253 second response time [16:42:04] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.969 second response time [16:42:15] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.077 second response time [16:42:19] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.322 second response time [16:42:44] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [16:44:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.50, 9.37, 10.05 [16:45:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.86, 9.90, 8.85 [16:45:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.34, 3.87, 3.98 [16:45:26] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.66, 1.58 [16:47:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.30, 9.38, 8.79 [16:47:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.68, 3.91 [16:47:26] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.25, 2.33, 1.83 [16:47:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.33, 10.08, 9.05 [16:49:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.54, 9.68, 9.02 [16:50:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.44, 3.96 [16:51:26] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.34, 1.74, 1.71 [16:52:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 3.89, 4.07 [16:52:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 10.97, 10.48 [16:52:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 11.38, 10.57 [16:53:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.40, 3.67 [16:53:26] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.18, 1.95, 1.79 [16:54:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.71, 3.98 [16:55:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.63, 3.72 [16:55:26] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.90, 1.89, 1.79 [16:56:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.34, 10.45, 10.48 [16:56:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.70, 10.70, 10.54 [16:57:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.68, 3.72 [16:59:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.94, 1.45, 1.64 [17:00:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.46, 10.77, 9.91 [17:00:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.78, 4.18, 4.09 [17:02:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.04, 11.91, 10.43 [17:02:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.66, 11.19, 10.72 [17:02:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 11.66, 10.99 [17:04:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.52, 11.12, 10.77 [17:05:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.89, 3.86 [17:05:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.92, 10.62, 10.23 [17:05:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.70, 3.93 [17:06:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 11.91, 11.26 [17:07:39] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [17:09:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.86, 6.45, 6.00 [17:09:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.34, 10.08, 9.55 [17:11:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.18, 5.98, 5.89 [17:11:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.58, 9.89, 10.03 [17:11:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:12:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.12, 3.91, 3.73 [17:13:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.97, 3.87 [17:14:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.00, 9.80, 10.18 [17:15:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 9.54, 9.63 [17:20:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.02, 11.29, 10.60 [17:20:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.72, 3.83 [17:20:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.54, 6.87, 6.28 [17:21:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.18, 10.59, 10.44 [17:22:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.69, 4.78, 4.20 [17:23:05] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.622 second response time [17:24:51] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.40, 7.37, 6.57 [17:26:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.93, 3.99 [17:26:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.13, 10.86 [17:26:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.07, 7.10, 6.59 [17:28:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.82, 11.08, 10.58 [17:29:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.57, 9.23, 8.11 [17:30:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.85, 9.71, 10.13 [17:30:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 10.44, 10.71 [17:30:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.85, 6.34, 6.38 [17:31:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.74, 9.28, 8.26 [17:32:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.18, 4.45, 4.12 [17:34:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 9.92, 8.74 [17:34:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.70, 10.36, 9.81 [17:35:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 10.61, 9.02 [17:36:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.99, 9.82, 8.85 [17:36:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7cac54f1ca7c [17:36:16] [02miraheze/mw-config] 07Universal-Omega 037cac54f - Only check if cache file exists in loadExtensions [17:36:18] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [17:36:18] [url] Page not found · GitHub · GitHub | github.com [17:36:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.05, 10.77, 10.01 [17:36:48] [02mw-config] 07Universal-Omega opened pull request 03#4756: Only check if cache file exists in loadExtensions - 13https://github.com/miraheze/mw-config/pull/4756 [17:36:49] ... [17:37:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.29, 10.05, 9.00 [17:38:19] miraheze/mw-config - Universal-Omega the build passed. [17:38:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.97, 10.87, 10.14 [17:38:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.35, 11.00 [17:38:49] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.77, 1.92, 1.48 [17:38:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4756: Only check if cache file exists in loadExtensions - 13https://github.com/miraheze/mw-config/pull/4756 [17:38:55] [url] Page not found · GitHub · GitHub | github.com [17:38:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7cac54f1ca7c...cbbf8a30ca88 [17:38:57] [url] Comparing 7cac54f1ca7c...cbbf8a30ca88 · miraheze/mw-config · GitHub | github.com [17:38:58] [02miraheze/mw-config] 07Universal-Omega 03cbbf8a3 - Update MirahezeFunctions.php [17:39:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:40:08] miraheze/mw-config - Universal-Omega the build passed. [17:40:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.80, 2.00 [17:40:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.82, 3.97 [17:40:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 10.77, 10.84 [17:40:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.91, 1.76, 1.46 [17:41:01] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [17:42:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.81, 3.94 [17:42:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.74, 11.04, 10.03 [17:42:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.34, 7.34, 6.87 [17:42:41] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.60, 1.65, 1.46 [17:42:58] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.21, 2.66 [17:44:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.70, 3.88 [17:44:58] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.53, 2.54, 2.48 [17:46:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.42, 4.83, 4.27 [17:47:53] [02mw-config] 07Universal-Omega closed pull request 03#4756: Only check if cache file exists in loadExtensions - 13https://github.com/miraheze/mw-config/pull/4756 [17:47:53] [url] Page not found · GitHub · GitHub | github.com [17:47:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fbca8a19badb...09a3b6c844df [17:47:55] [url] Comparing fbca8a19badb...09a3b6c844df · miraheze/mw-config · GitHub | github.com [17:47:56] [02miraheze/mw-config] 07Universal-Omega 0309a3b6c - Only check if cache file exists in loadExtensions (#4756) [17:47:57] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [17:47:59] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [17:47:59] ... [17:48:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.82, 2.03, 1.99 [17:48:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.46, 11.36, 10.69 [17:48:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.14, 6.69, 6.73 [17:48:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.81, 9.29, 10.13 [17:49:07] miraheze/mw-config - Universal-Omega the build passed. [17:49:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.50, 11.65, 10.89 [17:50:28] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 1.97, 1.65 [17:51:54] !log [@test101] starting deploy of {'config': True} to all [17:51:55] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [17:52:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.70, 9.92 [17:52:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.64, 10.19, 10.10 [17:52:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.74, 1.55, 1.54 [17:52:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.32, 3.79 [17:52:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:40] !log [@mwtask111] starting deploy of {'config': True} to all [17:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.39, 4.00 [17:53:57] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 17s [17:54:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.14, 11.54, 10.84 [17:55:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.52, 3.96 [17:56:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/61032841cb1a [17:56:17] [02miraheze/mw-config] 07Universal-Omega 036103284 - Update composer.json [17:56:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.87, 1.69 [17:56:19] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [17:56:19] [url] Page not found · GitHub · GitHub | github.com [17:56:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.91, 10.54, 10.31 [17:56:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.31, 3.63 [17:56:39] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.96, 1.73, 1.36 [17:57:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.59, 3.95 [17:58:14] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.94, 1.74 [17:58:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.00, 9.81, 10.08 [17:59:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.86, 1.96 [18:00:10] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.14, 1.67, 1.66 [18:00:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.76, 1.45 [18:01:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.09, 12.13, 11.27 [18:02:22] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.61, 1.68, 1.45 [18:02:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.33, 3.58 [18:02:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 10.83, 10.96 [18:04:05] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.98, 1.97, 1.80 [18:04:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.61, 10.20, 9.48 [18:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.83, 11.86, 11.30 [18:05:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 11.20, 11.15 [18:05:42] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.84, 1.91 [18:05:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 3.41, 3.68 [18:06:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.46, 8.90, 9.10 [18:08:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 11.33, 11.31 [18:09:53] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.10, 1.58, 1.70 [18:10:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:10:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.84, 3.69 [18:11:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.69, 10.87, 10.89 [18:11:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.94, 10.53, 9.28 [18:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.08, 10.22, 9.94 [18:12:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.55, 3.61 [18:13:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.92, 9.71, 9.13 [18:14:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.55, 9.78, 9.80 [18:15:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 11.85, 11.33 [18:15:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.74, 3.97 [18:16:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.70, 11.41, 10.71 [18:16:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.92, 2.95, 3.38 [18:18:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.35, 11.46, 11.02 [18:19:05] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.79, 11.50, 10.97 [18:21:14] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 9.498 second response time [18:21:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 8.78, 10.03 [18:21:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.31, 3.90 [18:22:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.25, 12.28, 11.33 [18:23:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:53] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.50, 3.89 [18:24:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 11.49, 11.17 [18:24:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.32, 10.68, 10.98 [18:24:35] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:31] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.050 second response time [18:27:39] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.902 second response time [18:27:44] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 7.556 second response time [18:27:52] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.515 second response time [18:28:01] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.792 second response time [18:28:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.312 second response time [18:29:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.67, 4.34, 3.87 [18:30:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.20, 8.92, 8.64 [18:31:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.75, 3.69 [18:32:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.79, 11.53, 11.16 [18:32:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.48, 8.91, 8.66 [18:32:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 10.84, 10.88 [18:33:26] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.87, 1.69 [18:33:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.22, 4.61, 4.01 [18:34:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.72, 3.44 [18:35:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.62, 1.40, 1.53 [18:36:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.49, 11.11, 11.14 [18:36:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.55, 4.43, 3.74 [18:36:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 11.14, 11.02 [18:39:26] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.09, 1.76, 1.71 [18:41:39] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:41:58] PROBLEM - wiki.chippyapps.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chippyapps.com All nameservers failed to answer the query. [18:43:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.55, 1.67 [18:44:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.75, 8.95, 10.20 [18:47:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.50, 9.40, 8.27 [18:49:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.52, 9.00, 8.24 [18:50:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.94, 11.18, 10.85 [18:51:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.70, 9.84, 9.18 [18:52:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.77, 11.21, 10.77 [18:52:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.68, 3.37, 3.93 [18:52:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 11.30, 10.93 [18:53:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.00, 9.56, 9.15 [18:54:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.91, 3.26, 2.65 [18:54:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.12, 4.11, 4.12 [18:56:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.87, 2.72, 2.53 [18:56:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.14, 11.31, 10.89 [18:56:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.77, 3.99 [18:58:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.93, 4.01 [18:59:03] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [18:59:03] [url] Page not found · GitHub · GitHub | github.com [18:59:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/13e40bbbde8d...97926356c2d4 [18:59:05] [url] Comparing 13e40bbbde8d...97926356c2d4 · miraheze/mw-config · GitHub | github.com [18:59:06] [02miraheze/mw-config] 07Universal-Omega 039792635 - Update MirahezeFunctions.php [19:00:12] miraheze/mw-config - Universal-Omega the build passed. [19:01:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.59, 10.56, 9.59 [19:03:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.54, 10.10, 9.54 [19:04:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 11.45, 11.12 [19:05:00] RECOVERY - wiki.chippyapps.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chippyapps.com' will expire on Tue 26 Jul 2022 05:48:03 GMT +0000. [19:06:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.13, 10.91, 11.20 [19:06:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 11.71, 11.28 [19:08:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.91, 10.43, 9.25 [19:09:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.95, 11.13, 10.02 [19:11:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.13, 10.10, 9.77 [19:14:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 9.87, 9.47 [19:16:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.17, 10.31, 10.57 [19:17:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.78, 9.99, 8.87 [19:18:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.78, 10.14, 10.45 [19:19:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.51, 8.80, 8.58 [19:20:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.48, 12.00, 11.10 [19:20:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.32, 3.89 [19:21:03] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/136fd943bf50 [19:21:05] [02miraheze/ManageWiki] 07Universal-Omega 03136fd94 - Fix reverting to default if default is empty string [19:21:06] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [19:21:07] [url] Page not found · GitHub · GitHub | github.com [19:21:08] [02ManageWiki] 07Universal-Omega opened pull request 03#362: Fix reverting to default if default is empty string - 13https://github.com/miraheze/ManageWiki/pull/362 [19:21:08] [url] Page not found · GitHub · GitHub | github.com [19:22:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.93, 11.36, 10.94 [19:22:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.52, 11.14 [19:23:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.62, 7.07, 6.36 [19:23:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.87, 3.02, 3.89 [19:24:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.68, 12.11, 11.26 [19:25:35] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.25, 6.46, 6.23 [19:25:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.40, 3.92 [19:26:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.29, 11.41, 11.11 [19:26:37] [02ManageWiki] 07Universal-Omega closed pull request 03#362: Fix reverting to default if default is empty string - 13https://github.com/miraheze/ManageWiki/pull/362 [19:26:37] [url] Page not found · GitHub · GitHub | github.com [19:26:38] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/65ac9c5351a7...4cd6dec22e81 [19:26:39] [url] Comparing 65ac9c5351a7...4cd6dec22e81 · miraheze/ManageWiki · GitHub | github.com [19:26:40] [02miraheze/ManageWiki] 07Universal-Omega 034cd6dec - Fix reverting to default if default is empty string (#362) [19:26:41] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:26:43] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [19:26:43] ... [19:26:50] miraheze/ManageWiki - Universal-Omega the build passed. [19:28:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.26, 11.40, 11.12 [19:29:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.74, 3.09, 3.73 [19:29:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.01, 3.83 [19:30:03] miraheze/ManageWiki - Universal-Omega the build passed. [19:30:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.55, 11.35, 11.16 [19:31:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.21, 3.97, 3.97 [19:31:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.56, 4.28, 4.19 [19:32:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.40, 3.50 [19:32:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.38, 10.81, 10.22 [19:32:49] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'robloxapi.wiki' expires in 15 day(s) (Thu 30 Jun 2022 19:06:45 GMT +0000). [19:34:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.37, 11.87, 11.35 [19:34:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.28, 9.41, 9.77 [19:34:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:03] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:27] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.90, 3.97 [19:36:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.75, 3.63 [19:36:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.87, 11.07, 11.55 [19:36:56] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.328 second response time [19:37:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.328 second response time [19:37:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/65ef188ddb8b...fedee6e86195 [19:37:04] [url] Comparing 65ef188ddb8b...fedee6e86195 · miraheze/ssl · GitHub | github.com [19:37:05] [02miraheze/ssl] 07MirahezeSSLBot 03fedee6e - Bot: Update SSL cert for robloxapi.wiki [19:37:26] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.026 second response time [19:37:55] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [19:38:07] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4cd6dec22e81...47711423846e [19:38:07] [url] Comparing 4cd6dec22e81...47711423846e · miraheze/ManageWiki · GitHub | github.com [19:38:08] [02miraheze/ManageWiki] 07Universal-Omega 034771142 - Fix [19:38:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.79, 11.34, 11.29 [19:39:10] PROBLEM - wiki.chippyapps.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.chippyapps.com could not be found [19:39:55] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [19:40:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.56, 12.62, 11.77 [19:40:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.43, 11.73, 11.57 [19:41:19] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:27] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:36] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:45] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:51] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:52] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:41:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.51, 3.94 [19:41:58] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [19:42:06] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:08] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.98, 11.63, 11.57 [19:42:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 4.84, 9.75, 10.92 [19:42:54] miraheze/ManageWiki - Universal-Omega the build passed. [19:43:19] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [19:43:25] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 7.335 second response time [19:43:31] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.921 second response time [19:43:36] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 3.377 second response time [19:43:48] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.009 second response time [19:43:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 6.051 second response time [19:43:51] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.021 second response time [19:43:53] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.882 second response time [19:43:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.39, 3.64 [19:44:02] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.024 second response time [19:44:05] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.864 second response time [19:44:09] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 8.157 second response time [19:44:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.96, 3.15, 3.39 [19:44:38] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.60, 13.88, 7.78 [19:44:39] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [19:44:45] [02mediawiki] 07Universal-Omega synchronize pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [19:44:46] [url] Page not found · GitHub · GitHub | github.com [19:44:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/886d578fd791...1c093427b678 [19:44:48] [url] Comparing 886d578fd791...1c093427b678 · miraheze/mediawiki · GitHub | github.com [19:44:48] [02miraheze/mediawiki] 07Universal-Omega 031c09342 - Update automerge.yml [19:45:01] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1074 bytes in 0.712 second response time [19:45:08] [02mediawiki] 07Universal-Omega synchronize pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [19:45:08] [url] Page not found · GitHub · GitHub | github.com [19:45:09] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1c093427b678...d51a1a984c02 [19:45:10] [url] Comparing 1c093427b678...d51a1a984c02 · miraheze/mediawiki · GitHub | github.com [19:45:11] [02miraheze/mediawiki] 07Universal-Omega 03d51a1a9 - Update automerge.yml [19:45:20] [discord] Is db121 down? [19:45:36] [discord] betawiki refuses to load [19:45:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.27, 2.92, 3.44 [19:46:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.61, 11.46, 11.30 [19:47:19] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [19:47:24] !sre [19:47:41] CosmicAlpha: hold deploys please [19:47:44] Agent: yes [19:47:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.41, 2.43, 3.20 [19:47:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.58, 2.48, 3.35 [19:47:57] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:58] [discord] Yeah, a user just reported issues too on Discord [19:47:59] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:09] [discord] I pinged the Site Reliability Engineers [19:48:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.57, 11.79, 11.44 [19:48:14] RhinosF1: not deploying anything right now. [19:48:17] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:20] paladox: Reception123: please look at db121 [19:48:32] CosmicAlpha: ok, please don't [19:48:32] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.30, 6.46, 6.08 [19:48:36] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:07] !log restart db121 mariadb (it OOM'd) [19:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:02] resolved [19:50:32] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 94 Threads: 113 Questions: 10159 Slow queries: 705 Opens: 1623 Open tables: 1617 Queries per second avg: 108.074 [19:50:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.16, 11.80, 11.55 [19:51:07] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32410 bytes in 6.609 second response time [19:51:15] Looks good here paladox [19:51:28] CosmicAlpha: feel free to deploy if needed now [19:52:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 12.21, 11.74 [19:53:55] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:07] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 5.619 second response time [19:54:14] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.818 second response time [19:54:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 4.072 second response time [19:54:36] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 1.746 second response time [19:54:57] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 3.185 second response time [19:55:44] [02mediawiki] 07Universal-Omega synchronize pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [19:55:44] [url] Page not found · GitHub · GitHub | github.com [19:55:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d51a1a984c02...b2f6f86258d8 [19:55:46] [url] Comparing d51a1a984c02...b2f6f86258d8 · miraheze/mediawiki · GitHub | github.com [19:55:47] [02miraheze/mediawiki] 07Universal-Omega 03b2f6f86 - Update automerge.yml [19:55:54] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 4.585 second response time [19:55:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.39, 3.37 [19:56:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.22, 3.73, 3.23 [19:56:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.54, 3.62, 3.32 [19:57:09] [02mediawiki] 07Universal-Omega closed pull request 03#7621: Automatically merge build only dependabot PRs - 13https://github.com/miraheze/mediawiki/pull/7621 [19:57:09] [url] Page not found · GitHub · GitHub | github.com [19:57:10] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f3b4663bb87b...e581e0d5a470 [19:57:11] [url] Comparing f3b4663bb87b...e581e0d5a470 · miraheze/mediawiki · GitHub | github.com [19:57:12] [02miraheze/mediawiki] 07Universal-Omega 03e581e0d - Automatically merge build only dependabot PRs (#7621) [19:57:13] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:57:15] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mediawiki [19:57:15] [url] Page not found · GitHub · GitHub | github.com [19:57:42] [02mediawiki] 07Universal-Omega labeled pull request 03#7626: Bump extensions/Cargo from `ed30677` to `751b633` - 13https://github.com/miraheze/mediawiki/pull/7626 [19:57:43] [url] Page not found · GitHub · GitHub | github.com [19:57:55] [02mediawiki] 07github-actions[bot] closed pull request 03#7626: Bump extensions/Cargo from `ed30677` to `751b633` - 13https://github.com/miraheze/mediawiki/pull/7626 [19:57:55] ... [19:57:56] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/e581e0d5a470...3a0aeb38ee0a [19:57:57] [url] Comparing e581e0d5a470...3a0aeb38ee0a · miraheze/mediawiki · GitHub | github.com [19:57:58] [02miraheze/mediawiki] 07dependabot[bot] 033a0aeb3 - Bump extensions/Cargo from `ed30677` to `751b633` (#7626) [19:57:59] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-751b633 [19:58:01] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-751b633 - 13https://github.com/miraheze/mediawiki [19:58:01] [url] Page not found · GitHub · GitHub | github.com [19:58:02] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7626: Bump extensions/Cargo from `ed30677` to `751b633` - 13https://github.com/miraheze/mediawiki/pull/7626 [19:58:02] [url] Page not found · GitHub · GitHub | github.com [19:58:34] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:11] [02mediawiki] 07Universal-Omega labeled pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [19:59:11] ... [19:59:15] CosmicAlpha: did everyone sign off on that? [20:00:05] If we're automerging build then we should auto pull [20:00:08] Just not auto sync [20:00:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.45, 11.91, 11.81 [20:00:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.36, 11.08, 11.65 [20:00:34] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.526 second response time [20:00:41] RhinosF1: I'm testing it. I plan to revert the change for a bit longer, but I wanted to see if it is working. Since it can only run if merged to REL1_38 due to pull_request_target, I had to merge to test. [20:00:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [20:01:22] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Mon 12 Sep 2022 18:36:58 GMT +0000. [20:01:47] CosmicAlpha: right [20:01:52] [02mediawiki] 07Universal-Omega unlabeled pull request 03#7622: Bump extensions/QuickInstantCommons from `14a7753` to `b07cab1` - 13https://github.com/miraheze/mediawiki/pull/7622 [20:01:52] [url] Page not found · GitHub · GitHub | github.com [20:01:54] Could you not have used a fork [20:02:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.77, 12.02, 11.85 [20:02:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.67, 11.91, 11.86 [20:02:38] CosmicAlpha: that logic is wrong anyway [20:02:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [20:02:44] It never checks for the build tag [20:02:50] Only that no others are there [20:03:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:45] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:48] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:50] RhinosF1: no, it checks that no others that it can ever automatically add are there. If it checked for build tag it would check that build exists whether the others do also or not. [20:03:52] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.222 second response time [20:04:03] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [20:04:19] !log DELETED and DROPPED https://phabricator.miraheze.org/P464 (PART 1) [20:04:19] [url] ✎ P464 Deleted wikis 14/06/2022 (PART 1) | phabricator.miraheze.org [20:04:20] CosmicAlpha: that doesn't account for a bug where none end up added [20:04:25] uh what happened? [20:04:33] Reception123: what? [20:04:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.02, 11.02, 11.58 [20:04:38] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:40] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.600 second response time [20:04:41] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:47] with db121 [20:05:02] Reception123: you're too late. Paladox fixed. [20:05:04] RhinosF1: you may be right. Either way I'll disable for now. [20:05:05] It OOM'd [20:05:12] CosmicAlpha: please [20:05:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:28] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:52] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 8.696 second response time [20:05:59] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 7.831 second response time [20:06:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 9.720 second response time [20:06:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.96, 9.95, 11.17 [20:06:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.72, 3.76 [20:06:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:32] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3a0aeb38ee0a...48ab61e82b76 [20:06:32] [url] Comparing 3a0aeb38ee0a...48ab61e82b76 · miraheze/mediawiki · GitHub | github.com [20:06:33] [02miraheze/mediawiki] 07Universal-Omega 0348ab61e - Disable build automerge for now [20:06:34] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:39] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 5.986 second response time [20:06:41] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 4.759 second response time [20:06:43] yeah I saw it was fixed though was wondering why [20:06:56] Reception123: OOM [20:07:01] yup saw now [20:07:03] also here's the status after deletion PART 1 [20:07:03] db101: 274G->284G [20:07:03] db111: 305G->314G [20:07:03] db121: 255G->266G [20:07:24] so around 30GB saved so far, not bad [20:07:25] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 9.474 second response time [20:07:28] RhinosF1: actually maybe automerging build is not necessarily, I did make it possible to mass merge by mass-adding the automerge label, maybe that is enough? [20:07:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.503 second response time [20:07:46] PROBLEM - wiki.chippyapps.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chippyapps.com All nameservers failed to answer the query. [20:07:58] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.044 second response time [20:08:03] You can just manually filter by label and mass apply the automerge label. [20:08:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.90, 3.81 [20:08:22] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23899 bytes in 2.587 second response time [20:08:32] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 0.313 second response time [20:09:39] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [20:10:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.39, 10.64, 11.11 [20:10:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.58, 10.61, 10.99 [20:10:41] [02mediawiki] 07Universal-Omega labeled pull request 03#7625: Bump extensions/Translate from `3b07a4b` to `f182386` - 13https://github.com/miraheze/mediawiki/pull/7625 [20:10:41] [url] Page not found · GitHub · GitHub | github.com [20:10:57] [02mediawiki] 07github-actions[bot] closed pull request 03#7625: Bump extensions/Translate from `3b07a4b` to `f182386` - 13https://github.com/miraheze/mediawiki/pull/7625 [20:10:58] [url] Page not found · GitHub · GitHub | github.com [20:10:59] [02miraheze/mediawiki] 07github-actions[bot] pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/48ab61e82b76...755112725a57 [20:11:00] [url] Comparing 48ab61e82b76...755112725a57 · miraheze/mediawiki · GitHub | github.com [20:11:00] [02miraheze/mediawiki] 07dependabot[bot] 037551127 - Bump extensions/Translate from `3b07a4b` to `f182386` (#7625) [20:11:02] [02miraheze/mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-f182386 [20:11:03] [02mediawiki] 07github-actions[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-f182386 - 13https://github.com/miraheze/mediawiki [20:11:04] [url] Page not found · GitHub · GitHub | github.com [20:11:05] [02mediawiki] 07github-actions[bot] unlabeled pull request 03#7625: Bump extensions/Translate from `3b07a4b` to `f182386` - 13https://github.com/miraheze/mediawiki/pull/7625 [20:11:05] [url] Page not found · GitHub · GitHub | github.com [20:11:52] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.25, 10.95, 11.17 [20:13:05] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.944 second response time [20:13:51] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.784 second response time [20:14:01] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 5.734 second response time [20:14:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.41, 11.38, 11.29 [20:14:24] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 6.682 second response time [20:14:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.81, 3.87 [20:14:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 11.26, 11.20 [20:15:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.67, 3.81 [20:16:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 10.62, 11.03 [20:18:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 4.03, 3.93 [20:19:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.36, 4.04, 3.88 [20:20:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.05, 10.78, 10.92 [20:20:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.97, 3.93 [20:22:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.63, 10.91, 10.95 [20:22:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 4.47, 4.12 [20:22:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 11.02, 10.97 [20:24:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 10.53, 10.83 [20:28:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.69, 9.10, 10.16 [20:31:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.06, 10.80, 9.03 [20:32:30] CosmicAlpha: if it works for you [20:32:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 10.97, 10.79 [20:32:41] Probably slightly better than the gomerge tool [20:33:11] I'm a lot more confident with semi automation [20:33:17] than I am full automation [20:33:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.56, 10.37, 9.10 [20:34:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.12, 10.90, 10.79 [20:35:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.28, 10.72, 9.35 [20:36:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.66, 9.63, 8.42 [20:36:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.44, 11.46, 11.01 [20:38:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.70, 9.52, 8.52 [20:38:23] RhinosF1: Will just leave it then. [20:38:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.46, 11.06 [20:38:36] (not automerge build) [20:39:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.11, 10.54, 9.66 [20:41:02] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 928 bytes in 1.510 second response time [20:41:19] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 15 minutes ago with 0 failures [20:42:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.40, 10.21, 10.12 [20:42:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.51, 10.21, 9.04 [20:44:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.08, 9.56, 9.90 [20:44:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.89, 9.73, 9.01 [20:48:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.10, 11.01, 10.87 [20:48:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.08, 6.43, 5.66 [20:49:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.88, 10.13, 10.01 [20:50:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.11, 11.32, 10.45 [20:50:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.59, 3.98 [20:50:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.59, 5.94, 5.57 [20:51:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.79, 1.93, 1.41 [20:52:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.12, 11.29, 10.55 [20:52:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 4.04, 4.10 [20:52:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.76, 11.66, 11.18 [20:53:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/97926356c2d4...0ebb13d8a91e [20:53:24] [url] Comparing 97926356c2d4...0ebb13d8a91e · miraheze/mw-config · GitHub | github.com [20:53:25] [02miraheze/mw-config] 07Universal-Omega 030ebb13d - Update LocalConfig.php [20:53:27] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [20:53:27] [url] Page not found · GitHub · GitHub | github.com [20:53:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.79, 1.42 [20:54:41] miraheze/mw-config - Universal-Omega the build passed. [20:56:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.09, 11.12, 10.58 [20:57:10] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32345 bytes in 4.850 second response time [20:57:50] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.46, 1.37 [20:58:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0ebb13d8a91e...2b023492a5fc [20:58:38] [url] Comparing 0ebb13d8a91e...2b023492a5fc · miraheze/mw-config · GitHub | github.com [20:58:39] [02miraheze/mw-config] 07Universal-Omega 032b02349 - Update buildConfigCache.php [20:58:41] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [20:58:41] ... [20:59:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2b023492a5fc...988f84b5a62b [20:59:31] [url] Comparing 2b023492a5fc...988f84b5a62b · miraheze/mw-config · GitHub | github.com [20:59:31] [02miraheze/mw-config] 07Universal-Omega 03988f84b - Update buildConfigCache.php [20:59:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [20:59:33] [url] Page not found · GitHub · GitHub | github.com [20:59:47] miraheze/mw-config - Universal-Omega the build passed. [21:00:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.61, 11.53, 10.90 [21:00:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.54, 3.90 [21:00:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/988f84b5a62b...daa9d755000f [21:00:32] [url] Comparing 988f84b5a62b...daa9d755000f · miraheze/mw-config · GitHub | github.com [21:00:32] [02miraheze/mw-config] 07Universal-Omega 03daa9d75 - Update buildConfigCache.php [21:00:34] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [21:00:34] [url] Page not found · GitHub · GitHub | github.com [21:00:52] miraheze/mw-config - Universal-Omega the build passed. [21:01:05] CosmicAlpha: ok [21:01:17] It might be worth having a proper conversation [21:01:26] Maybe when I've finished abstracting deploys [21:01:49] miraheze/mw-config - Universal-Omega the build passed. [21:02:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.59, 10.39, 10.12 [21:02:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 3.85, 3.96 [21:03:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/daa9d755000f...969c04191286 [21:03:09] [url] Comparing daa9d755000f...969c04191286 · miraheze/mw-config · GitHub | github.com [21:03:10] [02miraheze/mw-config] 07Universal-Omega 03969c041 - Update MirahezeFunctions.php [21:03:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [21:03:12] [url] Page not found · GitHub · GitHub | github.com [21:04:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.82, 10.27, 8.82 [21:04:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.85, 9.83, 9.95 [21:04:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.27, 11.54, 10.99 [21:04:21] miraheze/mw-config - Universal-Omega the build passed. [21:04:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.58, 3.86 [21:04:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/969c04191286...e80587035d68 [21:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.97, 11.41, 11.18 [21:04:34] [url] Comparing 969c04191286...e80587035d68 · miraheze/mw-config · GitHub | github.com [21:04:35] [02miraheze/mw-config] 07Universal-Omega 03e805870 - Update MirahezeFunctions.php [21:04:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [21:04:37] [url] Page not found · GitHub · GitHub | github.com [21:04:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e80587035d68...50a338e56ba3 [21:04:52] [url] Comparing e80587035d68...50a338e56ba3 · miraheze/mw-config · GitHub | github.com [21:04:53] [02miraheze/mw-config] 07Universal-Omega 0350a338e - Update buildConfigCache.php [21:04:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [21:04:55] [url] Page not found · GitHub · GitHub | github.com [21:05:41] miraheze/mw-config - Universal-Omega the build passed. [21:06:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.16, 9.35, 8.64 [21:06:05] miraheze/mw-config - Universal-Omega the build passed. [21:06:10] [02mw-config] 07Universal-Omega closed pull request 03#4753: Add method to build config cache for all wikis - 13https://github.com/miraheze/mw-config/pull/4753 [21:06:11] [url] Page not found · GitHub · GitHub | github.com [21:06:12] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [21:06:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [21:06:14] ... [21:07:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.40, 3.97 [21:08:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.74, 11.60, 11.22 [21:08:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 11.60, 11.35 [21:10:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.14, 9.99, 9.02 [21:10:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.26, 11.11, 9.95 [21:11:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.52, 3.87 [21:12:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.54, 10.39, 9.84 [21:13:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.42, 3.78 [21:14:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.47, 9.29, 10.20 [21:14:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.19, 2.42, 1.83 [21:14:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.65, 3.69 [21:14:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 10.79, 10.91 [21:14:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.63, 10.17, 9.83 [21:15:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.69, 3.84 [21:16:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 10.84, 10.94 [21:16:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.95, 9.48, 9.17 [21:17:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.77, 3.86 [21:18:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.94, 1.97, 1.76 [21:18:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.65, 3.71 [21:18:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.25, 10.71, 10.09 [21:20:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.17, 1.65, 1.67 [21:20:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 11.33, 11.06 [21:20:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.76, 9.68, 9.76 [21:21:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.23, 11.59, 10.85 [21:21:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.60, 3.77, 3.77 [21:22:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.81, 11.70, 10.98 [21:23:24] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 1.082 second response time [21:23:38] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:24:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.65, 3.64 [21:24:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.75, 11.97, 11.41 [21:26:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.52, 3.59 [21:26:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 12.27, 11.59 [21:26:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.07, 11.67, 11.11 [21:27:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.29, 10.10, 9.77 [21:28:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.93, 11.95, 11.29 [21:30:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.86, 4.46, 3.91 [21:30:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.79, 10.63, 11.21 [21:31:41] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 8.36, 2.81, 1.64 [21:31:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.82, 3.92 [21:32:29] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.05, 10.30, 9.75 [21:34:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.16, 9.82, 9.65 [21:34:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.51, 3.64 [21:34:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.31, 11.20, 11.20 [21:35:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.80, 1.63, 1.41 [21:36:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.98, 3.79 [21:36:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 10.43, 10.93 [21:37:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.50, 9.84, 9.91 [21:39:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:39:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.81, 4.91, 4.23 [21:40:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.78, 9.56, 8.60 [21:42:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.20, 9.41, 8.66 [21:42:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.43, 10.63, 10.65 [21:43:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.67, 3.87 [21:44:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.77, 3.95 [21:44:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 11.38, 10.96 [21:46:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.94, 3.98 [21:46:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.98, 12.07, 11.24 [21:48:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.40, 12.39, 11.24 [21:48:20] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:36] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.33, 3.58 [21:50:04] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.69, 3.86 [21:50:33] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:36] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:54] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 9.521 second response time [21:51:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.39, 1.43, 1.95 [21:52:02] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.686 second response time [21:52:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.78, 11.85, 11.34 [21:52:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 0.308 second response time [21:52:32] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.033 second response time [21:52:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 11.68, 11.50 [21:52:35] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.191 second response time [21:52:45] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.533 second response time [21:53:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.63, 1.96 [21:53:38] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:53:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 2.88, 3.35 [21:56:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.68, 10.04, 9.18 [21:56:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.51, 11.26, 11.14 [21:57:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.91, 2.00 [21:57:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.70, 9.73, 9.16 [21:58:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.74, 10.63, 10.92 [21:59:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.83, 3.62 [22:02:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.06, 2.95, 3.36 [22:02:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.92, 8.56, 10.02 [22:03:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.18, 10.72, 9.75 [22:03:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.22, 3.40 [22:03:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.31, 3.85 [22:04:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/61032841cb1a...836e9c4d9da2 [22:04:44] [url] Comparing 61032841cb1a...836e9c4d9da2 · miraheze/mw-config · GitHub | github.com [22:04:45] [02miraheze/mw-config] 07Universal-Omega 03836e9c4 - Update MirahezeFunctions.php [22:05:09] [02mw-config] 07Universal-Omega opened pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [22:05:10] [url] Page not found · GitHub · GitHub | github.com [22:05:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.67, 9.50, 9.42 [22:05:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.60, 3.88 [22:06:19] miraheze/mw-config - Universal-Omega the build has errored. [22:06:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.98, 10.75, 10.62 [22:07:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.72, 3.58 [22:09:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.73, 3.59 [22:10:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.35, 9.22, 10.07 [22:11:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.77, 3.61 [22:16:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.85, 9.00, 9.92 [22:16:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 10.41, 10.31 [22:19:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.02, 1.91 [22:19:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:19:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.29, 10.07, 9.75 [22:21:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.41, 10.00, 9.78 [22:21:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.38, 3.41 [22:22:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.28, 10.54, 10.33 [22:23:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.25, 3.17, 3.31 [22:26:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.53, 3.39 [22:27:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.70, 3.52 [22:29:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 3.80, 3.56 [22:30:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.77, 3.52 [22:31:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.68, 3.54 [22:32:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.31, 9.21, 9.92 [22:32:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.44, 3.44 [22:33:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.38, 1.58, 1.14 [22:33:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.94, 10.44, 10.17 [22:34:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.96, 3.26 [22:34:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.38, 11.36, 10.91 [22:35:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.63, 1.64, 1.22 [22:35:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.62, 3.54 [22:35:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.62, 3.95 [22:36:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.84, 11.54, 11.03 [22:39:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.51, 3.00, 1.94 [22:39:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.80, 11.53, 10.65 [22:39:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.49, 3.47 [22:39:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.66, 3.85 [22:41:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.18, 10.99, 10.55 [22:43:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.15, 2.00, 1.77 [22:43:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.78, 9.71, 10.13 [22:43:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.78, 3.56 [22:44:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.91, 3.52 [22:44:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 10.67, 10.73 [22:46:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.07, 3.61 [22:46:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 10.07, 10.51 [22:47:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.16, 1.61, 1.66 [22:47:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.72, 3.59 [22:48:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.30, 9.03, 10.07 [22:49:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.38, 4.36, 3.84 [22:51:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.89, 3.74 [22:52:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.76, 3.62 [22:52:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.09, 10.81, 9.85 [22:54:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.70, 3.07, 3.39 [22:54:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.20, 11.53, 10.77 [22:55:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.83, 1.94 [22:56:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.48, 10.66, 10.09 [22:56:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.47, 10.92, 10.66 [22:57:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.69, 3.60 [22:59:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.39, 3.50 [22:59:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.53, 3.93 [23:00:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.10, 9.99, 9.97 [23:00:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.96, 10.18, 9.26 [23:01:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.56, 3.88 [23:02:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.44, 10.02, 9.75 [23:02:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.38, 9.10, 8.96 [23:03:25] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:30] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.13, 9.59, 9.64 [23:04:03] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.48, 3.42 [23:04:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.88, 11.36, 10.88 [23:05:28] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 3.799 second response time [23:05:31] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15879 bytes in 2.606 second response time [23:05:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.89, 3.99 [23:06:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 0.324 second response time [23:06:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23898 bytes in 0.375 second response time [23:06:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.38, 3.05, 3.27 [23:06:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 11.15, 10.89 [23:06:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [23:07:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.31, 4.38, 4.15 [23:09:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.54, 4.69, 3.97 [23:10:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.03, 3.61 [23:12:07] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4289 MB (10% inode=96%); [23:12:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.85, 3.59 [23:13:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:13:55] [url] Page not found · GitHub · GitHub | github.com [23:13:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/836e9c4d9da2...51174cceac74 [23:13:57] [url] Comparing 836e9c4d9da2...51174cceac74 · miraheze/mw-config · GitHub | github.com [23:13:58] [02miraheze/mw-config] 07Universal-Omega 0351174cc - Update MirahezeFunctions.php [23:14:21] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:14:22] [url] Page not found · GitHub · GitHub | github.com [23:14:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/51174cceac74...147189a367be [23:14:23] [url] Comparing 51174cceac74...147189a367be · miraheze/mw-config · GitHub | github.com [23:14:24] [02miraheze/mw-config] 07Universal-Omega 03147189a - Update MirahezeFunctions.php [23:15:05] miraheze/mw-config - Universal-Omega the build has errored. [23:15:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/147189a367be...4620bb605a70 [23:15:28] [url] Comparing 147189a367be...4620bb605a70 · miraheze/mw-config · GitHub | github.com [23:15:28] [02miraheze/mw-config] 07Universal-Omega 034620bb6 - Update MirahezeFunctions.php [23:15:30] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:15:31] [url] Page not found · GitHub · GitHub | github.com [23:15:34] miraheze/mw-config - Universal-Omega the build has errored. [23:15:34] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.50, 1.68 [23:16:35] miraheze/mw-config - Universal-Omega the build passed. [23:16:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [23:17:41] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:17:42] [url] Page not found · GitHub · GitHub | github.com [23:17:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4620bb605a70...855f1ae328fe [23:17:43] [url] Comparing 4620bb605a70...855f1ae328fe · miraheze/mw-config · GitHub | github.com [23:17:44] [02miraheze/mw-config] 07Universal-Omega 03855f1ae - Update MirahezeFunctions.php [23:18:13] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/855f1ae328fe...9d16b3643511 [23:18:14] [url] Comparing 855f1ae328fe...9d16b3643511 · miraheze/mw-config · GitHub | github.com [23:18:15] [02miraheze/mw-config] 07github-actions 039d16b36 - CI: lint code to MediaWiki standards [23:18:16] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:18:17] [url] Page not found · GitHub · GitHub | github.com [23:18:48] miraheze/mw-config - Universal-Omega the build passed. [23:19:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9d16b3643511...dc47b60fb090 [23:19:24] [url] Comparing 9d16b3643511...dc47b60fb090 · miraheze/mw-config · GitHub | github.com [23:19:25] [02miraheze/mw-config] 07Universal-Omega 03dc47b60 - Update MirahezeFunctions.php [23:19:26] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:19:27] [url] Page not found · GitHub · GitHub | github.com [23:20:37] miraheze/mw-config - Universal-Omega the build passed. [23:21:01] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:21:01] [url] Page not found · GitHub · GitHub | github.com [23:21:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dc47b60fb090...190f1c89ce02 [23:21:03] [url] Comparing dc47b60fb090...190f1c89ce02 · miraheze/mw-config · GitHub | github.com [23:21:04] [02miraheze/mw-config] 07Universal-Omega 03190f1c8 - Update MirahezeFunctions.php [23:22:08] miraheze/mw-config - Universal-Omega the build passed. [23:23:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.84, 3.99 [23:24:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.04, 10.22, 9.39 [23:24:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:24:26] [url] Page not found · GitHub · GitHub | github.com [23:24:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/190f1c89ce02...61cec3ce5835 [23:24:27] [url] Comparing 190f1c89ce02...61cec3ce5835 · miraheze/mw-config · GitHub | github.com [23:24:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.90, 3.32 [23:24:28] [02miraheze/mw-config] 07Universal-Omega 0361cec3c - Update MirahezeFunctions.php [23:25:30] miraheze/mw-config - Universal-Omega the build passed. [23:25:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 4.24, 4.10 [23:26:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/61cec3ce5835...bfa8e5584b49 [23:26:41] [url] Comparing 61cec3ce5835...bfa8e5584b49 · miraheze/mw-config · GitHub | github.com [23:26:42] [02miraheze/mw-config] 07Universal-Omega 03bfa8e55 - Update MirahezeFunctions.php [23:26:44] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:26:44] [url] Page not found · GitHub · GitHub | github.com [23:27:19] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bfa8e5584b49...8cbb2c67ac4e [23:27:19] [url] Comparing bfa8e5584b49...8cbb2c67ac4e · miraheze/mw-config · GitHub | github.com [23:27:20] [02miraheze/mw-config] 07github-actions 038cbb2c6 - CI: lint code to MediaWiki standards [23:27:22] [02mw-config] 07github-actions[bot] synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:27:57] miraheze/mw-config - Universal-Omega the build passed. [23:27:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.06, 9.38, 9.30 [23:28:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.94, 3.64 [23:28:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.03, 9.65, 10.08 [23:32:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment with JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:32:40] [url] Page not found · GitHub · GitHub | github.com [23:32:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8cbb2c67ac4e...c0e3ffd0db86 [23:32:41] [url] Comparing 8cbb2c67ac4e...c0e3ffd0db86 · miraheze/mw-config · GitHub | github.com [23:32:42] [02miraheze/mw-config] 07Universal-Omega 03c0e3ffd - Update MirahezeFunctions.php [23:33:01] [02mw-config] 07Universal-Omega edited pull request 03#4757: Experiment using JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:33:01] [url] Page not found · GitHub · GitHub | github.com [23:33:50] miraheze/mw-config - Universal-Omega the build passed. [23:33:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.66, 3.94 [23:33:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.34, 3.89 [23:34:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.77, 3.66 [23:34:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.26, 10.14, 9.56 [23:34:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 11.07, 10.45 [23:35:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.98, 4.02, 4.05 [23:36:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.27, 9.47, 9.38 [23:36:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.61, 9.80, 10.07 [23:37:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.60, 3.90 [23:37:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 4.46, 4.24 [23:39:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.88, 3.95 [23:40:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.15, 4.09, 3.71 [23:42:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 10.35, 10.24 [23:44:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 9.59, 9.97 [23:46:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c0e3ffd0db86...55796e075ff2 [23:46:06] [url] Comparing c0e3ffd0db86...55796e075ff2 · miraheze/mw-config · GitHub | github.com [23:46:07] [02miraheze/mw-config] 07Universal-Omega 0355796e0 - Update composer.json [23:46:08] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment using JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:46:09] [url] Page not found · GitHub · GitHub | github.com [23:46:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 3.61, 3.70 [23:47:06] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [23:47:21] miraheze/mw-config - Universal-Omega the build passed. [23:49:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/55796e075ff2...9479be1d91c0 [23:49:22] [url] Comparing 55796e075ff2...9479be1d91c0 · miraheze/mw-config · GitHub | github.com [23:49:23] [02miraheze/mw-config] 07Universal-Omega 039479be1 - Update composer.json [23:49:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment using JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:49:25] [url] Page not found · GitHub · GitHub | github.com [23:50:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.60, 1.33 [23:50:37] miraheze/mw-config - Universal-Omega the build passed. [23:52:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9479be1d91c0...72679c365609 [23:52:04] [url] Comparing 9479be1d91c0...72679c365609 · miraheze/mw-config · GitHub | github.com [23:52:05] [02miraheze/mw-config] 07Universal-Omega 0372679c3 - Update composer.json [23:52:06] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [23:52:07] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment using JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:52:07] [url] Page not found · GitHub · GitHub | github.com [23:52:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.33, 10.17, 9.82 [23:52:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.71, 1.41 [23:53:10] miraheze/mw-config - Universal-Omega the build passed. [23:54:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.30, 9.48, 9.62 [23:54:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 3.95, 3.69 [23:54:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.18, 1.48, 1.36 [23:54:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/72679c365609...2d9f6e919108 [23:54:37] [url] Comparing 72679c365609...2d9f6e919108 · miraheze/mw-config · GitHub | github.com [23:54:38] [02miraheze/mw-config] 07Universal-Omega 032d9f6e9 - Update composer.json [23:54:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment using JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:54:40] [url] Page not found · GitHub · GitHub | github.com [23:55:41] miraheze/mw-config - Universal-Omega the build passed. [23:58:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.42, 3.57 [23:59:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2d9f6e919108...52c2fc49b664 [23:59:01] [url] Comparing 2d9f6e919108...52c2fc49b664 · miraheze/mw-config · GitHub | github.com [23:59:01] [02miraheze/mw-config] 07Universal-Omega 0352c2fc4 - Update composer.json [23:59:03] [02mw-config] 07Universal-Omega synchronize pull request 03#4757: Experiment using JsonReader - 13https://github.com/miraheze/mw-config/pull/4757 [23:59:03] [url] Page not found · GitHub · GitHub | github.com [23:59:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.42, 3.28, 3.84