[00:01:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.79, 1.85 [00:02:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:03:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.64, 1.77, 1.84 [00:05:56] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:56] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:32] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:36] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:12] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.70, 1.76 [00:07:50] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:08:37] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.440 second response time [00:08:39] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.643 second response time [00:09:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.60, 1.71 [00:09:51] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 2.294 second response time [00:10:01] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.325 second response time [00:10:03] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.577 second response time [00:11:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.30, 1.83, 1.78 [00:11:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.92, 1.97 [00:16:54] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.48, 2.72, 1.91 [00:17:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.28, 9.89, 9.37 [00:17:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 2.23, 2.07 [00:19:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.64, 9.37 [00:22:51] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.60, 1.77, 1.78 [00:23:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 10.18, 9.65 [00:25:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.63, 1.50, 1.94 [00:25:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.09, 9.60, 9.52 [00:26:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.64, 1.31, 1.59 [00:29:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.09, 1.67 [00:29:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.81, 1.98 [00:47:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.60, 2.15, 1.73 [00:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.93, 1.87 [00:49:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.46, 1.97, 1.72 [00:53:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.55, 1.61 [00:55:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.79, 10.65, 9.83 [00:57:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.60, 10.11, 9.72 [00:57:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.93, 1.97 [00:59:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.99, 2.00 [01:01:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.91, 1.97 [01:09:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.80, 1.86 [01:11:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.69, 1.81 [01:13:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.45, 3.95 [01:15:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.47, 5.13, 4.51 [01:17:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.62, 1.92, 1.83 [01:23:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.47, 3.95 [01:25:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 4.12, 4.14 [01:27:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.91, 1.96 [01:36:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.48, 3.99 [01:37:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.76, 1.82 [01:41:19] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.01, 6.84, 4.47 [01:41:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.47, 3.97 [01:44:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.18, 3.64, 3.79 [01:45:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.31, 3.98 [01:46:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.41, 3.70 [01:49:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.65, 3.92 [01:50:19] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 1.73, 1.37 [01:51:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.78, 3.94 [01:53:19] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.06, 7.72, 6.65 [01:53:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.27, 4.13, 4.04 [01:54:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.62, 1.45, 1.36 [01:54:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.77, 3.70 [01:55:19] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.31, 6.51, 6.34 [01:56:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.32, 3.55 [01:57:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.43, 4.06, 3.78 [01:58:14] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.74, 1.48 [01:59:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.77, 3.71 [02:01:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.44, 4.23, 3.87 [02:02:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.66, 1.79, 1.56 [02:03:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.79, 3.77 [02:03:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.91, 2.00 [02:04:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 4.03, 3.81 [02:05:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.75, 4.00 [02:06:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.87, 1.37, 1.45 [02:06:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.52, 3.64 [02:07:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.84, 4.00 [02:08:21] miraheze/SpriteSheet - Universal-Omega the build passed. [02:09:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 2.92, 3.38 [02:09:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.68, 3.91 [02:10:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.35, 2.99, 3.37 [02:11:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.79, 3.91 [02:13:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.59, 1.31 [02:14:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 3.99, 3.66 [02:15:24] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.45, 1.28 [02:15:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.73, 3.92 [02:18:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.69, 3.64 [02:19:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.96, 1.92 [02:20:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.55, 3.92, 3.50 [02:20:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 4.28, 3.87 [02:23:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.91, 4.03, 3.95 [02:24:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.71, 3.51 [02:25:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.67, 1.85, 1.57 [02:25:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.91, 9.78, 8.66 [02:25:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 4.21, 3.72 [02:26:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.82, 3.82 [02:27:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.56, 9.46, 8.67 [02:27:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.77, 3.61 [02:27:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.89, 1.95 [02:29:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.37, 1.44 [02:29:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.97, 3.99 [02:30:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 4.27, 4.00 [02:31:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.81, 4.02, 4.01 [02:32:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.77, 3.85 [02:32:56] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:02] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:13] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 6.128 second response time [02:33:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.61, 3.86 [02:33:40] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.67, 2.88, 3.33 [02:33:56] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.89, 1.91 [02:34:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.44, 10.44, 8.50 [02:34:56] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.338 second response time [02:35:00] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 2.899 second response time [02:35:15] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.462 second response time [02:35:35] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [02:35:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.74, 3.91, 3.93 [02:35:51] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [02:36:10] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.20, 9.74, 8.51 [02:37:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 10.38, 9.36 [02:37:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.64, 3.82 [02:37:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.94, 1.94 [02:39:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 3.89, 3.88 [02:39:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 2.08, 1.99 [02:40:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.71, 3.71 [02:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.98, 1.97 [02:43:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.68, 9.98, 9.52 [02:44:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.95, 4.08, 3.64 [02:44:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.78, 3.76 [02:46:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.76, 3.74 [02:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.58, 2.00, 1.95 [02:48:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.86, 3.66 [02:50:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.96, 3.72 [02:51:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.57, 3.60 [02:52:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.50, 9.58, 8.02 [02:53:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.93, 1.96 [02:54:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.75, 8.52, 7.83 [02:55:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 3.78, 3.64 [02:57:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.52, 3.56 [02:58:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.92, 4.00 [02:59:19] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.94, 5.71, 3.56 [03:01:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 2.92, 3.30 [03:03:01] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.55, 7.08, 6.20 [03:03:19] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.23, 6.10, 4.23 [03:04:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.39, 7.83, 6.57 [03:05:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.66, 3.99 [03:08:47] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.64, 7.05, 6.62 [03:10:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.67, 3.76 [03:10:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.12, 6.20, 6.37 [03:12:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.56, 3.71 [03:13:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 3.44, 3.65 [03:14:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.69, 3.73 [03:15:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.04, 3.48 [03:16:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.06, 3.15, 3.53 [03:17:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.48, 2.44, 3.20 [03:17:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.41, 1.95, 1.83 [03:19:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.95, 1.10 [03:20:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 2.96, 3.39 [03:20:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.76, 3.41 [03:21:13] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ttafwikiwiki --no-updates --username-prefix=wikia:talkingfriends /mnt/mediawiki-static/metawiki/ImportDump/ttafwikiwiki-20220627204154.xml (END - exit=0) [03:21:14] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ttafwikiwiki (START) [03:21:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.91, 1.54, 1.05 [03:25:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.93, 1.90 [03:26:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.02, 4.04, 3.65 [03:26:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.05, 4.51, 3.73 [03:27:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 2.02, 1.93 [03:28:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.12, 3.28, 3.43 [03:29:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.72, 4.01, 3.69 [03:30:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.55, 3.52 [03:30:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.24, 3.39 [03:31:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.91, 3.68 [03:37:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.16, 2.92, 3.34 [03:37:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.87, 1.94 [03:38:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.43, 2.82, 3.30 [03:41:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.45, 3.46 [03:41:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.91, 1.93 [03:43:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.87, 1.91 [03:45:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.54, 1.73, 1.33 [03:45:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.82, 3.56 [03:47:17] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.48, 1.29 [03:47:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.45, 3.18, 3.38 [03:47:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.91, 3.63 [03:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.95, 2.31, 2.07 [03:49:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.92, 3.66 [03:51:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.56, 3.56 [03:53:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.08, 3.38 [03:57:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.82, 1.97 [03:59:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.86, 1.97 [04:01:55] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 13.01, 8.61, 6.03 [04:01:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.85, 1.95 [04:03:27] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [04:03:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.50, 2.06, 2.02 [04:03:59] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:01] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:26] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 0.438 second response time [04:05:56] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.316 second response time [04:05:59] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [04:06:11] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [04:07:42] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [04:07:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.86, 1.95 [04:09:42] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [04:11:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.51, 6.90, 6.94 [04:13:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.40, 6.24, 6.70 [04:16:43] SRE, `[1ace6f723c8e739e195e1042] 2022-06-28 04:16:24: Fatal exception of type "TypeError"` [04:17:21] Looking [04:17:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.80, 2.12, 1.96 [04:18:56] Argument 1 passed to MediaWiki\Extension\DiscordNotifications\Hooks::getDiscordUserText() must implement interface MediaWiki\User\UserIdentity, instance of MediaWiki\Extension\CentralAuth\User\CentralAuthGroupMembershipProxy given, called in /srv/mediawiki/w/extensions/DiscordNotifications/includes/Hooks.php on line 474 [04:19:01] my extension also... [04:24:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1c2e42868b09...2a9470c46df9 [04:24:32] [url] Comparing 1c2e42868b09...2a9470c46df9 · miraheze/mediawiki · GitHub | github.com [04:24:32] [02miraheze/mediawiki] 07Universal-Omega 032a9470c - Update DiscordNotifications [04:24:34] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [04:24:42] dmehus: ^ [04:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:27:56] !log [universalomega@test101] starting deploy of {'world': True} to all [04:27:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.82, 1.95 [04:28:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:29:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 2.03, 2.01 [04:30:10] CosmicAlpha, oh interesting [04:31:05] It's interesting it was a DiscordNotifications extension bug, yet it didn't produce a gblrights log log entry [04:31:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.30, 2.93 [04:31:44] I guess that makes sense, though, depending on the order in which events happened [04:31:52] * dmehus looking at your commit now [04:33:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.15, 3.07, 2.87 [04:33:29] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 535s [04:33:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:36] dmehus: it caused fatal in a hook execution, so never got to do anything at all. Should be fixed now. [04:34:12] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 376s [04:34:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:45:27] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.69, 7.41, 6.66 [04:45:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.63, 1.93 [04:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.68, 1.99, 2.03 [04:50:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.71, 3.12 [04:51:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.40, 3.97, 3.29 [04:52:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.45, 3.10 [04:53:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.64, 3.25 [04:54:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.39, 3.39, 3.12 [04:55:04] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.10, 7.48, 7.09 [04:55:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.55, 4.00, 3.41 [04:57:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.58, 3.33 [04:58:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.82, 6.44, 6.73 [04:59:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.65, 2.90, 3.11 [04:59:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.78, 1.97 [05:01:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.16, 3.05 [05:03:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.06, 3.03 [05:15:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.93, 1.87 [05:21:48] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=ttafwikiwiki (END - exit=0) [05:21:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:30:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.58, 3.33 [05:34:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.10, 4.22, 3.61 [05:34:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.76, 3.20 [05:35:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.77, 1.96 [05:36:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.72, 3.51 [05:36:37] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mrjaroslavikwiki --summary=Imported from psp.miraheze.org --u=Miraheze_maintenance_script --search-recursively /home/reception/mrjaroslavik (START) [05:36:38] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=mrjaroslavikwiki --summary=Imported from psp.miraheze.org --u=Miraheze_maintenance_script --search-recursively /home/reception/mrjaroslavik (END - exit=256) [05:36:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:36:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.40, 3.14 [05:36:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.22, 3.34 [05:39:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.84, 1.93 [05:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.77, 1.90 [05:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.88, 1.89 [05:51:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 2.00, 1.94 [05:57:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.77, 3.38 [06:01:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.60, 3.41 [06:03:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 3.16, 3.26 [06:05:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.64, 2.11, 1.95 [06:10:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 10.53, 9.24 [06:12:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 10.26, 9.30 [06:13:00] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rollbackEdits.php --wiki=webkinzguidewiki --user=Miraheze_maintenance_script (END - exit=65280) [06:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 9.38, 9.10 [06:28:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.67, 3.25 [06:32:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.67, 3.31 [06:33:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.37, 3.18 [06:33:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.33, 3.34, 3.22 [06:35:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.70, 3.33 [06:37:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.35, 3.25 [06:39:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 9.89, 9.41 [06:41:26] PROBLEM - microworldwiki.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'microworldwiki.ga' expires in 15 day(s) (Thu 14 Jul 2022 06:29:57 GMT +0000). [06:41:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.88, 9.59, 9.36 [06:41:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ef09da82f7c9...1079fe962ab7 [06:41:43] [url] Comparing ef09da82f7c9...1079fe962ab7 · miraheze/ssl · GitHub | github.com [06:41:44] [02miraheze/ssl] 07MirahezeSSLBot 031079fe9 - Bot: Update SSL cert for microworldwiki.ga [06:42:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.69, 3.40 [06:43:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.71, 1.37 [06:43:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.51, 1.09 [06:44:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.74, 3.40, 3.33 [06:45:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.39, 1.29 [06:45:03] PROBLEM - wiki.jkrstudios.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jkrstudios.org' expires in 15 day(s) (Thu 14 Jul 2022 06:28:51 GMT +0000). [06:45:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.58, 1.18, 1.03 [06:51:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1079fe962ab7...7076c1511cde [06:51:06] [url] Comparing 1079fe962ab7...7076c1511cde · miraheze/ssl · GitHub | github.com [06:51:07] [02miraheze/ssl] 07MirahezeSSLBot 037076c15 - Bot: Update SSL cert for wiki.jkrstudios.org [06:52:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.58, 3.44, 3.35 [06:54:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.24, 3.29 [07:09:54] PROBLEM - nottingham.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nottingham.wiki' expires in 15 day(s) (Thu 14 Jul 2022 07:06:09 GMT +0000). [07:10:22] RECOVERY - microworldwiki.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'microworldwiki.ga' will expire on Mon 26 Sep 2022 05:41:37 GMT +0000. [07:16:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7076c1511cde...80b114d000a9 [07:16:54] [url] Comparing 7076c1511cde...80b114d000a9 · miraheze/ssl · GitHub | github.com [07:16:55] [02miraheze/ssl] 07MirahezeSSLBot 0380b114d - Bot: Update SSL cert for nottingham.wiki [07:31:05] PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wilderyogi.eu' expires in 15 day(s) (Thu 14 Jul 2022 07:26:31 GMT +0000). [07:35:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/80b114d000a9...77e51b1b6f97 [07:35:18] [url] Comparing 80b114d000a9...77e51b1b6f97 · miraheze/ssl · GitHub | github.com [07:35:18] [02miraheze/ssl] 07MirahezeSSLBot 0377e51b1 - Bot: Update SSL cert for wiki.wilderyogi.eu [07:38:48] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [07:44:09] RECOVERY - wiki.jkrstudios.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jkrstudios.org' will expire on Mon 26 Sep 2022 05:51:00 GMT +0000. [07:47:56] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.56, 3.89, 2.77 [07:49:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.94, 2.88, 2.53 [08:00:09] RECOVERY - wiki.wilderyogi.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wilderyogi.eu' will expire on Mon 26 Sep 2022 06:35:10 GMT +0000. [08:25:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 1.77, 1.29 [08:27:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.42, 1.72, 1.34 [08:29:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 4.06, 3.29 [08:29:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.72, 1.45, 1.30 [08:31:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.36, 3.47, 3.17 [08:33:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.26, 3.13 [08:48:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 4.02, 3.48 [08:50:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.73, 3.44 [08:52:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 3.24, 3.30 [08:56:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.80, 3.49 [08:58:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.69, 3.14 [09:00:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.80, 3.56 [09:04:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.17, 3.35 [09:04:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.77, 3.92, 3.38 [09:05:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.09, 1.85, 1.44 [09:05:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.77, 3.51, 2.98 [09:07:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.76, 1.86, 1.50 [09:07:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.60, 3.08 [09:07:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.62, 3.53 [09:09:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.36, 1.64, 1.46 [09:09:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.79, 3.51 [09:11:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 3.14, 3.30 [09:13:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.92, 3.28, 3.12 [09:15:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.65, 2.75, 3.22 [09:29:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.75, 1.99 [09:31:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.93, 1.88, 2.01 [09:35:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.93, 1.99 [09:39:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.91, 1.96 [09:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.91, 1.95 [09:44:29] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 4.11, 2.44, 1.12 [09:46:29] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 0.94, 1.78, 1.04 [09:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.32, 1.93, 1.92 [09:48:29] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.43, 1.29, 0.95 [09:53:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.96, 1.98 [10:07:45] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.95, 2.06, 1.65 [10:08:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.15, 2.90 [10:10:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.17, 2.92 [10:12:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 2.91, 2.87 [10:13:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.93, 1.74 [10:13:59] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.59, 1.70 [10:15:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.53, 1.92, 1.75 [10:17:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.50, 1.70, 1.68 [10:19:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 1.94, 1.80 [10:23:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.90, 1.83 [10:25:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 2.11, 1.92 [10:33:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.98, 1.95 [10:39:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.89, 1.89 [10:41:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 2.88, 2.49 [10:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.85, 1.88 [10:43:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.29, 2.90, 2.53 [10:43:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.89, 1.90 [10:45:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.85, 1.89 [10:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.46, 2.06, 1.96 [10:53:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.90, 1.94 [11:07:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.10, 3.99, 3.09 [11:09:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.80, 3.13 [11:11:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.83, 3.76, 3.17 [11:13:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.45, 3.12 [11:14:41] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.57, 1.74, 1.31 [11:15:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.95, 2.94, 2.98 [11:16:41] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.58, 1.59, 1.30 [11:17:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 2.01, 1.86 [11:19:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 9.99, 8.77 [11:22:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 3.87, 3.06 [11:23:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.37, 9.60, 8.88 [11:24:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.30, 2.94 [11:25:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 2.00, 1.97 [11:33:32] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.51, 3.35, 2.18 [11:38:13] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.84, 1.52 [11:38:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.35, 2.91 [11:39:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.69, 1.77 [11:40:07] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.87, 1.48, 1.42 [11:40:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.59, 3.24 [11:41:28] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.37, 1.79, 1.86 [11:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.35, 1.55, 1.71 [11:42:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.05, 2.91 [11:42:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 3.04, 3.08 [11:43:59] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.58, 1.58, 1.70 [11:45:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.81, 1.28, 1.63 [11:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.80, 2.06, 1.86 [11:50:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.87, 3.40 [11:52:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.41, 3.43, 3.30 [11:54:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.28, 3.22, 3.23 [11:59:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.92, 1.98 [12:09:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.76, 2.05, 1.94 [12:25:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 2.21, 1.71 [12:27:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.09, 1.94, 1.68 [12:30:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 3.70, 2.99 [12:32:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.69, 3.07 [12:33:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.76, 1.39, 1.52 [12:33:42] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:33:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.76, 1.99 [12:36:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 3.10, 2.97 [12:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.91, 1.87 [12:48:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.10, 3.24 [12:52:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 9.97, 9.24 [12:52:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.60, 3.24 [12:54:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.93, 3.39 [12:56:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.82, 3.33 [12:56:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.47, 3.29 [12:57:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.50, 3.23 [12:58:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.64, 3.32 [12:58:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.10, 3.17 [12:59:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.05, 3.10 [13:00:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.92, 3.11, 3.17 [13:02:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:03:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.88, 1.98 [13:04:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.40, 3.27 [13:04:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.32, 3.22 [13:06:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.02, 3.12 [13:08:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.32, 3.27 [13:11:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.61, 3.42 [13:15:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.34, 10.17, 10.17 [13:16:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.47, 3.26 [13:17:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.38, 4.31, 3.75 [13:17:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.78, 1.84 [13:18:31] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 5.32, 3.07, 1.37 [13:18:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.51, 3.29 [13:18:49] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.70, 3.32, 2.09 [13:19:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 10.60, 10.37 [13:21:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.93, 3.70 [13:22:29] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.25, 1.48, 1.09 [13:22:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.56, 3.37 [13:22:47] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.84, 1.94, 1.80 [13:23:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 4.16, 3.80 [13:25:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.10, 9.84, 10.13 [13:25:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.88, 1.93 [13:26:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.68, 3.40 [13:27:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.03, 1.81, 1.55 [13:28:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 4.18, 3.62 [13:28:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.05, 1.54, 1.67 [13:29:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.64, 1.52 [13:30:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.83, 3.56 [13:30:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.84, 3.58 [13:33:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.81, 4.00 [13:34:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.24, 3.37 [13:34:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.80, 3.61 [13:34:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.44, 7.43, 6.68 [13:35:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.29, 4.52, 4.20 [13:36:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.16, 5.05, 4.12 [13:36:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.14, 6.82, 6.54 [13:37:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.77, 1.81 [13:38:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.97, 3.67 [13:38:13] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.05, 2.25, 1.79 [13:40:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.98, 3.71 [13:40:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.77, 6.14, 6.35 [13:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.76, 1.81 [13:42:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.65, 1.91, 1.75 [13:42:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 4.17, 3.81 [13:45:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.14, 1.56, 1.65 [13:46:52] !log removed dmehus from stewards@ [13:46:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:47:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.73, 3.77 [13:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.05, 2.09, 1.89 [13:48:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.81, 4.00 [13:50:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.94, 4.01 [13:51:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 10.24, 9.67 [13:51:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 4.41, 4.00 [13:52:15] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.21, 6.50, 6.17 [13:54:11] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.65, 6.25, 6.12 [13:54:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.54, 3.87 [13:54:42] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=celebswiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/celebswiki-20220628135356.xml (START) [13:54:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.11, 9.95, 9.70 [13:56:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.71, 3.89 [13:57:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.96, 1.78 [13:59:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.62, 1.81, 1.75 [14:00:14] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=celebswiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/celebswiki-20220628135916.xml (START) [14:00:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:35] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wiqwiki --no-updates --username-prefix=wikia:wiq /mnt/mediawiki-static/metawiki/ImportDump/wiqwiki-20220627161915.xml (START) [14:00:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:01:27] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.90, 1.69, 1.50 [14:01:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.45, 1.62 [14:02:37] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=celebswiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/celebswiki-20220628135356.xml (END - exit=0) [14:02:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.91, 3.96 [14:02:38] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=celebswiki (START) [14:02:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.96, 1.38, 1.40 [14:03:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.27, 10.71, 9.67 [14:04:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 4.11, 4.01 [14:06:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.25, 10.52, 9.79 [14:06:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:06:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.53, 3.81 [14:07:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.49, 3.86 [14:08:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.56, 10.49, 9.87 [14:08:48] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.27, 7.70, 6.86 [14:09:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.61, 3.99 [14:10:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.61, 7.84, 7.02 [14:11:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.62, 3.85, 4.05 [14:12:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.48, 10.41 [14:13:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 4.22, 4.02 [14:13:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 10.35, 10.12 [14:15:32] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.35, 9.39, 9.79 [14:15:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.25, 9.93, 9.98 [14:16:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.27, 6.77, 6.78 [14:17:27] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wiqwiki --no-updates --username-prefix=wikia:wiq /mnt/mediawiki-static/metawiki/ImportDump/wiqwiki-20220627161915.xml (END - exit=0) [14:17:28] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wiqwiki (START) [14:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:17:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:19:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.67, 3.88 [14:20:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.19, 4.20, 3.83 [14:21:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.33, 4.10 [14:25:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=wiqwiki (END - exit=0) [14:25:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:29:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.88, 2.00 [14:31:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.62, 1.71, 1.40 [14:31:56] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 7.18, 4.45, 2.48 [14:33:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.97, 1.43, 1.33 [14:33:42] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [14:36:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 10.72, 10.15 [14:37:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.54, 3.98 [14:38:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=celebswiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/celebswiki-20220628135916.xml (END - exit=0) [14:38:18] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=celebswiki (START) [14:38:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:38:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:40:44] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/77e51b1b6f97...3dfe31ec83ea [14:40:45] [url] Comparing 77e51b1b6f97...3dfe31ec83ea · miraheze/ssl · GitHub | github.com [14:40:46] [02miraheze/ssl] 07Reception123 033dfe31e - add privacywiki.xyz cert [14:41:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.84, 10.32 [14:42:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.39, 6.75, 6.38 [14:43:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.64, 3.82 [14:44:05] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.41, 1.73, 1.57 [14:44:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.97, 6.08, 6.18 [14:45:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.18, 2.97, 3.54 [14:46:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.51, 1.51 [14:47:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.15, 4.16, 3.88 [14:47:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 11.57, 10.81 [14:47:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 1.98, 1.89 [14:49:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.75, 3.76 [14:50:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.75, 7.77, 6.79 [14:52:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.78, 7.40, 6.78 [14:54:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.94, 6.78, 6.62 [14:54:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:57:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.69, 3.65 [14:57:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.16, 11.63, 11.07 [14:57:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.95, 1.98 [15:01:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.07, 3.85 [15:01:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 11.70, 11.25 [15:02:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [15:02:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.86, 4.11, 4.15 [15:05:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 11.39, 11.17 [15:05:03] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.62, 7.05, 6.80 [15:05:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.82, 3.77 [15:06:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.37, 3.84 [15:06:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 11.44, 11.22 [15:08:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.55, 3.84 [15:09:13] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.46, 1.69 [15:09:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.52, 4.18, 3.87 [15:10:53] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.47, 6.78, 6.78 [15:12:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.71, 10.42, 9.50 [15:14:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.44, 10.61, 9.90 [15:14:48] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.26, 7.64, 7.09 [15:15:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.76, 1.74 [15:16:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.76, 11.87, 11.44 [15:16:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.29, 9.75, 9.47 [15:17:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.01, 2.10, 1.86 [15:18:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.92, 11.50, 10.43 [15:18:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.57, 3.88 [15:18:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.22, 11.74, 11.46 [15:18:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.85, 7.61, 7.24 [15:20:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 11.43, 10.52 [15:20:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.99, 12.23, 11.67 [15:20:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.85, 7.82, 7.33 [15:21:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.73, 3.87 [15:21:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.19, 11.10, 9.59 [15:21:30] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:33] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:39] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [15:21:45] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:48] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.53, 10.80, 10.40 [15:22:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.64, 7.27, 7.21 [15:23:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.44, 10.66, 9.61 [15:25:00] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:16] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:22] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:41] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:44] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:26:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 11.79, 11.71 [15:26:27] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.37, 6.19, 6.78 [15:28:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 12.05, 11.82 [15:28:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.44, 2.58, 3.30 [15:30:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.29, 3.60 [15:30:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 11.53, 11.66 [15:30:43] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:31:23] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.30, 11.66, 10.39 [15:31:54] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.918 second response time [15:31:55] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 3.01, 3.47 [15:32:07] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.809 second response time [15:32:09] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.630 second response time [15:32:11] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.753 second response time [15:32:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.79, 11.03, 10.56 [15:32:25] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 4.460 second response time [15:32:27] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [15:32:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.93, 2.00 [15:32:45] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:33:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.29, 11.75, 10.60 [15:34:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 2.81, 3.34 [15:34:41] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:35:03] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=celebswiki (END - exit=35072) [15:35:07] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset -0.002406775951 secs [15:35:11] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 40 minutes ago with 0 failures [15:35:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:20] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 1.097 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [15:35:28] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [15:36:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.43, 9.68, 8.61 [15:36:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.40, 11.61, 10.95 [15:36:27] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [15:36:39] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [15:36:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.94, 1.98 [15:37:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.78, 9.86, 10.16 [15:38:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.68, 8.26, 8.22 [15:38:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.71, 3.43 [15:39:18] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 27096 bytes in 0.670 second response time [15:39:56] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.307 second response time [15:39:56] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.026 second response time [15:39:57] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.021 second response time [15:40:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.83, 3.63 [15:40:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.36, 11.51, 11.02 [15:40:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.48, 3.39 [15:40:34] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.316 second response time [15:42:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.05, 4.36, 3.83 [15:42:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.91, 10.86, 10.84 [15:42:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.67, 4.40, 3.71 [15:43:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.41, 10.70, 9.85 [15:43:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.18, 11.31 [15:45:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.84, 11.37, 11.36 [15:46:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.60, 3.55 [15:47:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.31, 9.75, 9.67 [15:47:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.70, 12.18, 11.66 [15:48:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.74, 3.73 [15:48:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.73, 3.59 [15:48:57] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.79, 11.11, 10.39 [15:50:03] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.87, 9.14, 10.07 [15:50:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.53, 3.53 [15:50:35] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:37] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.85, 11.22, 10.52 [15:54:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.74, 2.80, 3.24 [15:56:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 2.74, 3.27 [15:58:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.78, 3.57 [15:58:58] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.442 second response time [15:59:00] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.054 second response time [15:59:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.20, 9.66, 10.14 [15:59:29] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.274 second response time [15:59:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 6.653 second response time [16:00:24] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.371 second response time [16:00:47] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.43, 7.54, 6.43 [16:02:44] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.68, 6.83, 6.31 [16:03:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.41, 3.93 [16:03:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.07, 10.61, 9.92 [16:04:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.46, 10.45, 9.13 [16:04:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.33, 10.73, 10.04 [16:04:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 4.02, 3.69 [16:04:40] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.33, 6.67, 6.31 [16:05:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.65, 10.19, 10.15 [16:06:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 10.75, 9.42 [16:06:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.33, 10.37, 9.98 [16:06:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.72, 3.61 [16:06:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.63, 1.95 [16:07:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.31, 9.19, 9.77 [16:08:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.25, 10.09, 9.33 [16:08:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.12, 4.02, 3.72 [16:09:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 4.08, 4.04 [16:10:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.38, 3.24 [16:10:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.70, 3.64 [16:10:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.88, 1.96 [16:11:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/74370cb317f3 [16:11:29] [02miraheze/mw-config] 07Universal-Omega 0374370cb - Add initSiteStats.php to ImportDumpScriptCommand [16:11:30] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [16:11:31] [url] Page not found · GitHub · GitHub | github.com [16:11:32] [02mw-config] 07Universal-Omega opened pull request 03#4801: Add initSiteStats.php to ImportDumpScriptCommand - 13https://github.com/miraheze/mw-config/pull/4801 [16:11:32] [url] Page not found · GitHub · GitHub | github.com [16:12:02] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.92, 1.46 [16:12:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.62, 10.11, 10.02 [16:12:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.78, 1.92 [16:12:54] miraheze/mw-config - Universal-Omega the build passed. [16:13:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.85, 3.97 [16:14:02] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.84, 1.50, 1.35 [16:14:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.80, 3.67 [16:15:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.89, 4.16, 4.08 [16:16:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.66, 3.62 [16:18:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.89, 3.71 [16:18:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.54, 2.07, 1.98 [16:20:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.98, 3.85 [16:20:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.50, 11.58, 10.64 [16:20:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.69, 10.75, 9.95 [16:22:06] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.02, 10.68, 10.02 [16:22:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.65, 3.68 [16:24:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.39, 9.84, 9.79 [16:24:09] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.537 second response time [16:24:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.28, 11.22, 10.74 [16:25:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.64, 4.00 [16:26:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.43, 3.65, 3.62 [16:27:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.52, 9.89, 10.18 [16:27:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 4.21, 4.17 [16:27:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.64, 10.33, 9.99 [16:28:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.58, 1.34 [16:28:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.89, 1.97 [16:30:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.93, 3.74 [16:30:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.17, 1.42, 1.32 [16:30:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.45, 11.74, 11.03 [16:30:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.40, 3.48, 3.61 [16:31:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.68, 3.97 [16:32:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.82, 3.73 [16:32:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:32:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.08, 10.11, 10.21 [16:33:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.37, 9.44, 9.79 [16:34:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.20, 11.79, 11.25 [16:34:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.16, 2.86, 3.34 [16:36:03] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/3bbe540936eb [16:36:04] [02miraheze/MirahezeMagic] 07Universal-Omega 033bbe540 - Update MirahezeMagicHooks.php [16:36:06] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/MirahezeMagic [16:36:06] [url] Page not found · GitHub · GitHub | github.com [16:36:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:18] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:36:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.16, 10.78, 10.41 [16:36:53] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3bbe540936eb...dfec9c79fb2d [16:36:54] [url] Comparing 3bbe540936eb...dfec9c79fb2d · miraheze/MirahezeMagic · GitHub | github.com [16:36:55] [02miraheze/MirahezeMagic] 07Universal-Omega 03dfec9c7 - Update MirahezeMagicLogEmailManager.php [16:36:59] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:14] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/dfec9c79fb2d...f929b72c0a21 [16:37:14] [url] Comparing dfec9c79fb2d...f929b72c0a21 · miraheze/MirahezeMagic · GitHub | github.com [16:37:15] [02miraheze/MirahezeMagic] 07Universal-Omega 03f929b72 - Update extension.json [16:37:18] CosmicAlpha: would it be better to make mwscript just do it? [16:37:23] Rather than a super long command [16:37:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.84, 4.01, 3.93 [16:38:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.58, 3.59 [16:38:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.71, 12.17, 11.52 [16:38:19] [02MirahezeMagic] 07Universal-Omega opened pull request 03#349: Remove support for MediaWiki versions before 1.38 - 13https://github.com/miraheze/MirahezeMagic/pull/349 [16:38:19] [url] Page not found · GitHub · GitHub | github.com [16:38:21] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.538 second response time [16:38:22] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.827 second response time [16:38:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.16, 10.34, 10.29 [16:39:00] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.752 second response time [16:40:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.27, 3.47 [16:40:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.43, 11.60, 11.40 [16:41:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.93, 3.92 [16:42:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.13, 11.11, 10.58 [16:43:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/00d869cb7aba [16:44:00] [02miraheze/mw-config] 07Universal-Omega 0300d869c - Remove (centralauth-)usermerge rights [16:44:02] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [16:44:02] [url] Page not found · GitHub · GitHub | github.com [16:44:04] [02mw-config] 07Universal-Omega opened pull request 03#4802: Remove (centralauth-)usermerge rights - 13https://github.com/miraheze/mw-config/pull/4802 [16:44:04] ... [16:44:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.91, 3.68 [16:44:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.63, 11.60, 11.42 [16:44:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.06, 11.04, 10.62 [16:44:49] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=celebswiki (END - exit=35072) [16:45:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:29] miraheze/mw-config - Universal-Omega the build passed. [16:45:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.91, 3.89 [16:46:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.77, 3.67 [16:46:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.80, 1.79 [16:46:57] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:47:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.47, 3.41 [16:47:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.64, 3.80 [16:47:54] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.90, 7.07, 6.23 [16:48:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.87, 11.74, 11.53 [16:49:50] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.76, 7.08, 6.33 [16:51:47] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.53, 7.46, 6.55 [16:51:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 10.67, 11.83 [16:52:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.91, 3.20 [16:53:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.46, 1.18 [16:55:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.34, 1.17 [16:55:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.85, 4.63, 4.07 [16:56:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.99, 3.79 [16:56:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.91, 1.92 [16:57:58] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.45, 1.38, 3.68 [16:58:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.85, 3.25, 3.55 [16:58:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 10.71, 10.45 [16:58:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.99, 1.94 [16:59:58] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.55, 1.09, 3.29 [17:00:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 3.65, 3.64 [17:00:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 10.74, 10.51 [17:00:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.87, 1.90 [17:01:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.93, 3.99 [17:01:36] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:02:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.26, 12.15, 11.49 [17:02:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.51, 9.64, 10.15 [17:02:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.93, 1.92 [17:03:00] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/667238c53bd0...7b0944aff5fd [17:03:01] [url] Comparing 667238c53bd0...7b0944aff5fd · miraheze/MirahezeMagic · GitHub | github.com [17:03:02] [02miraheze/MirahezeMagic] 07Universal-Omega 037b0944a - Remove support for MediaWiki versions before 1.38 (#349) [17:03:03] [02MirahezeMagic] 07Universal-Omega closed pull request 03#349: Remove support for MediaWiki versions before 1.38 - 13https://github.com/miraheze/MirahezeMagic/pull/349 [17:03:04] [url] Page not found · GitHub · GitHub | github.com [17:03:05] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [17:03:06] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/MirahezeMagic [17:03:07] [url] Page not found · GitHub · GitHub | github.com [17:03:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.92, 7.89, 7.32 [17:04:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.67, 3.67 [17:04:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.86, 1.90 [17:05:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.08, 11.80, 11.57 [17:07:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.67, 1.97, 1.55 [17:07:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 10.90, 11.28 [17:08:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.88, 3.04, 3.38 [17:09:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.66, 1.81, 1.54 [17:10:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.95, 1.92 [17:11:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.93, 11.47, 10.66 [17:12:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.85, 1.89 [17:12:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.50, 10.69, 9.80 [17:13:07] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.39, 7.51, 7.23 [17:13:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.29, 1.58, 1.51 [17:13:48] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:15:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.12, 11.52, 10.94 [17:15:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 2.73, 3.27 [17:16:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.57, 9.92, 9.72 [17:17:00] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.17, 7.25, 7.23 [17:18:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.41, 10.62, 11.62 [17:18:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 2.02, 1.93 [17:18:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.90, 1.50 [17:19:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.97, 9.33, 10.18 [17:20:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.59, 1.43 [17:20:53] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.05, 6.01, 6.72 [17:22:39] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.15, 3.94, 2.32 [17:23:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.02, 4.39, 3.58 [17:23:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.17, 5.30, 4.14 [17:23:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.88, 5.93, 4.35 [17:23:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.87, 9.88, 10.26 [17:30:09] @Owen In your opinion, should T&S be entitled to moderation flags in #miraheze? We weren’t sure. [17:30:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.90, 1.93 [17:37:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.18, 11.62, 10.88 [17:38:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.34, 1.63 [17:39:23] [discord] T&S doesn’t extend to IRC and Discord [17:39:44] Reception123: ^ mind doing the flags? [17:39:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 11.52, 10.96 [17:39:59] thanks @Owen [17:40:47] And in -ops too please Reception123 [17:42:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.73, 11.11, 10.66 [17:44:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.44, 11.06, 10.68 [17:44:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.58, 9.93, 8.94 [17:46:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.16, 11.68, 11.21 [17:46:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.94, 9.27, 8.80 [17:48:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.72, 11.22, 11.09 [17:49:15] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.32, 11.39, 11.15 [17:50:15] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.29, 1.93, 1.62 [17:52:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.50, 1.74, 1.56 [17:52:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.74, 1.59 [17:53:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 6.910 second response time [17:53:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.46, 11.11 [17:54:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.554 second response time [17:54:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.48, 1.55, 1.51 [17:55:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.71, 11.27 [17:56:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.36, 10.88, 11.10 [17:56:44] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.464 second response time [17:57:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.61, 1.59 [17:57:26] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.70, 2.69, 1.87 [17:59:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.54, 9.92, 9.38 [17:59:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 11.65, 11.31 [18:00:01] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.72, 1.98 [18:00:50] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:03:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.75, 9.55, 9.37 [18:04:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.51, 12.07, 11.53 [18:04:14] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 4.756 second response time [18:04:57] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.054 second response time [18:05:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.15, 1.73, 1.76 [18:06:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.02, 11.46, 11.38 [18:06:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.82, 9.69, 10.14 [18:06:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 2.07, 2.03 [18:09:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.77, 1.33, 1.60 [18:09:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.78, 12.00, 11.72 [18:10:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.06, 11.61, 11.47 [18:10:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.03, 10.44, 10.37 [18:12:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 10.61, 11.12 [18:12:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.11, 11.05, 10.60 [18:13:21] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.25, 10.54, 10.49 [18:16:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.53, 11.20, 11.23 [18:17:53] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.82, 11.25, 11.25 [18:18:18] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:18:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.28, 11.42, 10.79 [18:18:49] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.78, 1.70 [18:20:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 10.22, 9.67 [18:20:38] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.32, 11.20, 10.80 [18:20:46] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.88, 10.05, 9.66 [18:26:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 10.29, 9.91 [18:26:38] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.85, 2.30, 1.95 [18:27:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.09, 9.90, 9.81 [18:27:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 11.46, 11.33 [18:28:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.89, 9.16, 10.18 [18:28:35] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.49, 1.91, 1.85 [18:29:04] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 9.982 second response time [18:29:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [18:29:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 11.12, 11.22 [18:30:03] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.486 second response time [18:31:09] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.201 second response time [18:31:10] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.966 second response time [18:32:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.54, 10.99, 10.64 [18:32:27] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.036 second response time [18:32:30] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.31, 2.55, 2.08 [18:34:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.00, 10.80, 10.64 [18:34:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.26, 9.69, 10.18 [18:35:34] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:24] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.00, 1.82, 1.90 [18:36:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.64, 11.45 [18:38:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.96, 11.29, 10.82 [18:38:24] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.65, 1.78, 1.27 [18:38:51] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.578 second response time [18:39:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 11.62, 11.46 [18:40:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.27, 10.66, 10.66 [18:40:18] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.83, 1.87 [18:40:21] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.49, 1.22 [18:40:47] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.981 second response time [18:41:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.61, 11.89, 11.57 [18:42:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.02, 10.90, 10.44 [18:42:49] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.855 second response time [18:43:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.13, 11.81, 11.60 [18:44:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.263 second response time [18:44:10] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 4.296 second response time [18:44:13] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.94, 1.93 [18:44:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.44, 10.56, 10.36 [18:45:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.75, 6.72, 6.12 [18:46:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.12, 9.63, 10.20 [18:48:09] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.11, 1.69, 1.81 [18:48:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 9.73, 10.10 [18:49:44] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.13, 6.14, 6.04 [18:49:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 12.28, 11.87 [18:50:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.37, 1.62, 1.77 [18:50:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.28, 10.25, 10.31 [18:51:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.52, 10.82, 9.75 [18:51:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.23, 11.72, 11.72 [18:52:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.37, 10.76, 10.48 [18:53:15] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 10.37, 9.72 [18:53:38] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.94, 12.11, 11.86 [18:54:09] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.13, 1.76, 1.79 [18:54:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.42, 10.86, 10.55 [18:54:44] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.87, 9.62, 9.53 [18:58:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.81, 1.53, 1.71 [18:58:32] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.49, 7.41, 6.66 [18:59:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [18:59:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 11.41, 11.65 [19:00:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.51, 1.52, 1.67 [19:00:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.75, 11.27, 10.76 [19:00:28] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.94, 7.29, 6.71 [19:01:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.86, 11.79 [19:02:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 8.013 second response time [19:02:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.283 second response time [19:03:34] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.862 second response time [19:03:35] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.109 second response time [19:04:03] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.700 second response time [19:04:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.19, 11.81, 11.14 [19:04:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.71, 6.44, 6.49 [19:05:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.54, 11.65 [19:10:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.71, 11.74, 11.24 [19:11:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.36, 11.33, 10.58 [19:12:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.62, 11.54, 11.23 [19:14:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 11.87, 11.39 [19:15:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 10.27, 9.35 [19:16:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.27, 9.24 [19:17:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.34, 9.97, 9.35 [19:20:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 9.49, 9.20 [19:20:09] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.34, 1.89, 1.51 [19:21:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.53, 10.28, 9.55 [19:22:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.16, 1.78, 1.42 [19:22:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.60, 1.69, 1.48 [19:22:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.48, 11.85, 11.76 [19:22:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.64, 10.39, 10.95 [19:23:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.28, 9.95, 9.52 [19:24:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.06, 1.63, 1.42 [19:24:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.38, 6.95, 6.49 [19:24:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 2.30, 1.68 [19:25:34] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:50] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.80, 10.69 [19:26:09] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.01, 2.31, 1.78 [19:26:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.19, 11.40, 11.54 [19:26:19] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:21] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.38, 10.31, 9.82 [19:27:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.59, 10.31, 10.55 [19:27:57] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.94, 6.48, 6.44 [19:28:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.97, 1.71 [19:29:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.69, 9.56, 9.60 [19:29:46] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.458 second response time [19:29:53] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.877 second response time [19:29:56] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.144 second response time [19:30:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.75, 1.67 [19:30:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.89, 11.60, 11.60 [19:30:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.53, 1.58 [19:30:30] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.168 second response time [19:30:36] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27089 bytes in 6.903 second response time [19:32:09] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.46, 1.95, 1.74 [19:32:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.60, 12.41, 11.90 [19:33:48] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.020 second response time [19:34:05] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.16, 1.58, 1.63 [19:35:51] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.359 second response time [19:35:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 11.84, 11.19 [19:36:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 10.28, 9.59 [19:36:13] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.552 second response time [19:36:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.57, 11.70, 11.14 [19:37:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.36, 11.52, 11.15 [19:38:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.42, 9.86, 9.53 [19:38:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.76, 1.62 [19:38:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.51, 11.14, 11.01 [19:40:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.54, 1.56 [19:40:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.28, 11.40, 11.76 [19:42:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.01, 11.89, 11.31 [19:44:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.00, 11.78, 11.80 [19:46:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.11, 11.31, 11.64 [19:46:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.72, 10.87, 11.07 [19:50:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.01, 11.78, 11.76 [19:50:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.45, 10.01, 9.63 [19:51:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.77, 6.80, 6.35 [19:52:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.33, 11.63, 11.71 [19:52:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.44, 9.79, 9.60 [19:52:48] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:16] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.55, 6.69, 6.37 [19:53:16] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:29] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:32] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:54:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.406 second response time [19:55:18] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.983 second response time [19:55:34] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 6.301 second response time [19:55:37] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.477 second response time [19:55:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.77, 11.52, 11.10 [19:56:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.86, 11.41, 11.53 [19:56:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.23, 10.32, 9.88 [19:58:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.66, 11.09, 11.41 [19:58:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.31, 9.40, 10.16 [19:59:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.84, 11.94, 11.39 [20:00:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.14, 10.11, 9.89 [20:01:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 12.01, 11.47 [20:02:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.74, 11.63, 11.50 [20:03:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.78, 11.44, 11.32 [20:04:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 10.43, 10.37 [20:05:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.59, 12.15, 11.59 [20:07:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 11.90, 11.57 [20:08:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.63, 12.00, 11.77 [20:09:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 12.16, 11.70 [20:10:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.83, 12.33, 11.92 [20:10:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.63, 3.06, 1.83 [20:10:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.54, 11.61, 10.83 [20:11:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 11.62, 11.56 [20:15:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.17, 10.95, 11.52 [20:15:05] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.39, 1.64, 1.53 [20:15:05] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.63, 8.52, 7.13 [20:15:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.39, 10.71, 10.72 [20:15:10] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:15] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:19] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:57] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.25, 11.36, 11.59 [20:16:38] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.56, 7.34, 6.86 [20:17:07] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.679 second response time [20:17:15] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.522 second response time [20:17:19] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.422 second response time [20:17:55] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 4.115 second response time [20:20:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.15, 9.93, 9.60 [20:20:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.96, 11.47, 11.58 [20:20:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.19, 11.11, 10.84 [20:22:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.18, 9.60, 9.52 [20:22:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.31, 10.93, 10.81 [20:23:29] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:14] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:25:27] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.165 second response time [20:26:11] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 3.635 second response time [20:27:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 10.86, 11.13 [20:28:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.18, 11.45, 11.49 [20:29:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.77, 9.90, 10.76 [20:30:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.67, 11.30, 11.44 [20:30:21] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.28, 7.62, 7.18 [20:30:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.39, 9.38, 10.07 [20:32:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.39, 12.43, 11.83 [20:32:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.44, 7.78, 7.31 [20:35:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 11.20, 11.01 [20:37:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.73, 11.05, 10.99 [20:39:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.87, 11.45, 11.12 [20:40:21] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.26, 7.34, 7.20 [20:41:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.51, 11.19 [20:42:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.32, 6.64, 6.97 [20:42:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.00, 11.41, 10.56 [20:44:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.80, 11.39, 11.82 [20:44:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.81, 6.19, 6.78 [20:44:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 11.15, 10.57 [20:50:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.30, 11.26, 11.59 [20:50:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.72, 6.73, 6.78 [20:52:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.95, 10.89, 11.42 [20:52:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.25, 6.58, 6.72 [20:56:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.62, 1.33 [20:58:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.44, 10.74, 11.06 [20:58:38] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.06, 1.40, 1.28 [20:59:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.78, 11.29 [21:01:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.66, 2.11, 1.58 [21:01:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.99, 10.49, 10.88 [21:05:35] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.37, 1.83, 1.38 [21:05:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.91, 1.62 [21:05:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.07, 10.99 [21:06:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.52, 11.84, 11.57 [21:07:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.30, 1.69, 1.38 [21:07:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.60, 1.54 [21:07:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 10.78, 10.92 [21:10:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 9.25, 9.97 [21:14:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.06, 10.14, 10.23 [21:16:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.10, 11.15, 11.18 [21:18:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.74, 10.80, 11.06 [21:18:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.82, 9.89, 10.12 [21:19:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.75, 10.60 [21:20:13] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 4.11, 3.31, 1.62 [21:21:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.03, 10.71 [21:22:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.58, 11.22, 11.13 [21:24:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.31, 10.05, 10.71 [21:24:13] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.24, 1.57, 1.28 [21:24:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.49, 3.45, 3.92 [21:27:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 11.53, 11.00 [21:29:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.49, 10.39, 10.22 [21:29:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 10.52, 10.70 [21:31:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.37, 9.37, 9.88 [21:34:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [21:34:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 3.98, 3.96 [21:35:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 10.79, 10.65 [21:37:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.37, 10.52, 10.56 [21:38:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.68, 11.59, 10.95 [21:39:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.97, 10.71 [21:40:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 11.16, 10.87 [21:40:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.87, 3.92 [21:42:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.48, 4.37, 4.11 [21:43:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.47, 10.58, 10.07 [21:43:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 11.46, 10.97 [21:45:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.75, 9.97, 9.92 [21:46:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.80, 11.43, 11.02 [21:48:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.60, 11.03, 10.93 [21:49:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.72, 10.79, 10.33 [21:51:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 11.11, 10.86 [21:52:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.87, 11.59, 11.16 [21:52:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.42, 9.99, 10.09 [21:54:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 10.34, 10.76 [21:55:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 11.60, 11.14 [21:58:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.65, 3.98 [22:02:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.60, 3.96 [22:02:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.62, 11.08, 10.70 [22:04:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.14, 10.83, 10.64 [22:04:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.38, 9.95, 9.81 [22:05:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.83, 11.45 [22:06:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [22:06:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.43, 9.82, 9.78 [22:06:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.76, 1.96 [22:07:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.31, 10.92, 11.14 [22:08:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.39, 6.63, 5.94 [22:10:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.80, 3.77 [22:10:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.96, 1.99 [22:12:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.75, 3.76 [22:12:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.96, 10.74, 10.17 [22:12:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.85, 10.45, 9.41 [22:13:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.38, 3.88 [22:14:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.12, 4.00 [22:14:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.05, 6.73, 6.25 [22:14:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/cc8e979425a3 [22:14:31] [02miraheze/mw-config] 07Universal-Omega 03cc8e979 - Remove extra check for getManageWikiConfigCache [22:14:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [22:14:33] [url] Page not found · GitHub · GitHub | github.com [22:14:37] [02mw-config] 07Universal-Omega opened pull request 03#4803: Remove extra check for getManageWikiConfigCache - 13https://github.com/miraheze/mw-config/pull/4803 [22:14:37] ... [22:14:42] [02mw-config] 07Universal-Omega edited pull request 03#4803: Remove extra check for getManageWikiConfigCache - 13https://github.com/miraheze/mw-config/pull/4803 [22:14:43] ... [22:14:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.95, 1.98 [22:14:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.98, 9.81, 9.30 [22:14:46] [02mw-config] 07Universal-Omega edited pull request 03#4803: Remove extra check for getManageWikiConfigCache - 13https://github.com/miraheze/mw-config/pull/4803 [22:14:46] [url] Page not found · GitHub · GitHub | github.com [22:15:15] [02mw-config] 07Universal-Omega edited pull request 03#4803: Remove extra check for getManageWikiConfigCache - 13https://github.com/miraheze/mw-config/pull/4803 [22:15:16] [url] Page not found · GitHub · GitHub | github.com [22:15:25] [02mw-config] 07Universal-Omega labeled pull request 03#4803: Remove extra check for getManageWikiConfigCache - 13https://github.com/miraheze/mw-config/pull/4803 [22:15:25] ... [22:15:44] miraheze/mw-config - Universal-Omega the build passed. [22:16:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.96, 3.95 [22:16:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.01, 11.08, 10.43 [22:16:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 2.00, 2.00 [22:17:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 3.74, 3.88 [22:17:40] [02mw-config] 07Universal-Omega closed pull request 03#4803: Remove extra check for getManageWikiConfigCache - 13https://github.com/miraheze/mw-config/pull/4803 [22:17:40] [url] Page not found · GitHub · GitHub | github.com [22:17:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/249b4fd85262...a53445be5d9e [22:17:42] [url] Comparing 249b4fd85262...a53445be5d9e · miraheze/mw-config · GitHub | github.com [22:17:43] [02miraheze/mw-config] 07Universal-Omega 03a53445b - Remove extra check for getManageWikiConfigCache (#4803) [22:17:44] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [22:17:45] [url] Page not found · GitHub · GitHub | github.com [22:17:46] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [22:18:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 4.22, 4.05 [22:18:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.54, 11.16, 10.54 [22:18:45] miraheze/mw-config - Universal-Omega the build passed. [22:19:22] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [22:19:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:45] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [22:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.76, 3.90 [22:20:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 2.92, 3.39 [22:20:43] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [22:20:47] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 3s [22:20:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.50, 3.75 [22:21:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.29, 11.28, 10.96 [22:23:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.96, 10.13, 10.59 [22:24:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.29, 9.27, 9.97 [22:26:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 4.22, 4.06 [22:26:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.77, 3.59 [22:27:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 4.06, 3.91 [22:28:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a0718e20c563 [22:28:34] [02miraheze/mw-config] 07Universal-Omega 03a0718e2 - Don't read entire cache twice [22:28:35] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [22:28:36] [url] Page not found · GitHub · GitHub | github.com [22:28:37] [02mw-config] 07Universal-Omega opened pull request 03#4804: Don't read entire cache twice - 13https://github.com/miraheze/mw-config/pull/4804 [22:28:37] ... [22:28:52] !log [universalomega@test101] starting deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-6', 'config': True, 'ignoretime': True} to all [22:28:54] !log [universalomega@test101] finished deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-6', 'config': True, 'ignoretime': True} to all - SUCCESS in 2s [22:28:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:51] miraheze/mw-config - Universal-Omega the build passed. [22:31:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.70, 10.70, 10.34 [22:31:48] [02mw-config] 07Universal-Omega labeled pull request 03#4804: Don't read entire cache twice - 13https://github.com/miraheze/mw-config/pull/4804 [22:31:48] [url] Page not found · GitHub · GitHub | github.com [22:32:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.92, 11.04, 10.53 [22:33:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.84, 3.92 [22:34:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a53445be5d9e...3cfa636b10c3 [22:34:10] [url] Comparing a53445be5d9e...3cfa636b10c3 · miraheze/mw-config · GitHub | github.com [22:34:11] [02miraheze/mw-config] 07Universal-Omega 033cfa636 - Don't read entire cache twice (#4804) [22:34:13] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [22:34:14] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [22:34:15] [url] Page not found · GitHub · GitHub | github.com [22:34:16] [02mw-config] 07Universal-Omega closed pull request 03#4804: Don't read entire cache twice - 13https://github.com/miraheze/mw-config/pull/4804 [22:34:16] ... [22:34:17] !log [universalomega@test101] starting deploy of {'pull': 'config', 'branch': 'master', 'config': True, 'ignoretime': True} to all [22:34:21] !log [universalomega@test101] finished deploy of {'pull': 'config', 'branch': 'master', 'config': True, 'ignoretime': True} to all - SUCCESS in 3s [22:34:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:31] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [22:34:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:48] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [22:35:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:35:14] miraheze/mw-config - Universal-Omega the build passed. [22:36:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.84, 3.18, 3.70 [22:40:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 3.75, 3.79 [22:41:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.51, 4.33, 4.00 [22:42:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 9.26, 8.89 [22:43:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.66, 9.56, 10.15 [22:44:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.99, 3.90 [22:44:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.86, 8.97, 8.83 [22:46:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 4.38, 4.07 [22:46:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.49, 9.22, 10.00 [22:47:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.81, 3.93 [22:47:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.90, 1.48 [22:49:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.92, 3.95 [22:49:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.85, 1.50 [22:49:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.51, 10.29 [22:50:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.38, 11.08, 10.65 [22:50:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.67, 1.80, 1.29 [22:51:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.60, 3.83 [22:52:13] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 4.13, 2.27, 1.12 [22:52:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.97, 10.53, 10.49 [22:52:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.71, 1.32 [22:53:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.69, 4.75, 4.23 [22:53:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.63, 1.51 [22:54:13] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.89, 1.64, 1.02 [22:54:46] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.43, 1.65, 1.35 [22:59:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 11.13, 10.93 [23:00:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.37, 9.86, 10.18 [23:01:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.92, 11.79, 11.19 [23:02:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.56, 3.94 [23:03:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.86, 10.39, 10.20 [23:03:43] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.19, 5.30, 3.55 [23:05:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.98, 9.95, 10.06 [23:06:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.53, 3.88 [23:06:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 3.65, 3.86 [23:06:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:10] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:31] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.14, 3.65 [23:08:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 27088 bytes in 5.382 second response time [23:09:12] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.675 second response time [23:09:33] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.997 second response time [23:09:43] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.41, 6.00, 4.52 [23:10:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 4.03, 4.00 [23:11:43] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.08, 5.99, 4.69 [23:12:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.62, 3.85 [23:12:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.64, 3.74 [23:13:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.91, 10.20, 10.22 [23:13:43] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.42, 5.83, 4.80 [23:14:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.86, 3.90 [23:14:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.47, 3.66 [23:14:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.73, 1.99 [23:15:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.56, 11.16, 10.50 [23:15:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.27, 10.90, 9.49 [23:15:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.89, 9.55, 9.99 [23:16:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.67, 3.84 [23:16:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.21, 3.91, 3.79 [23:16:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [23:16:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.51, 1.92, 2.02 [23:16:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.03, 11.07, 10.55 [23:17:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.37, 10.17, 9.41 [23:17:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.74, 11.89 [23:18:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.85, 3.89 [23:18:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.52, 3.65 [23:19:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.20, 10.63, 9.32 [23:20:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.87, 3.75 [23:21:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.49, 10.96, 9.61 [23:23:43] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.16, 5.07, 4.92 [23:25:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 11.99, 11.92 [23:26:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.44, 3.61, 3.75 [23:28:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.67, 3.91 [23:28:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.53, 11.49, 10.78 [23:29:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.37, 9.85, 9.63 [23:29:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.49, 11.58, 11.82 [23:30:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.83, 1.97 [23:31:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.28, 3.96 [23:32:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.96, 2.67, 3.30 [23:34:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.66, 11.00, 10.88 [23:38:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.16, 3.65, 3.65 [23:39:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.46, 3.75 [23:40:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.12, 9.31, 10.12 [23:41:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.45, 3.72 [23:42:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.40, 3.58 [23:42:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.70, 3.47 [23:43:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.61, 3.74 [23:44:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.51, 3.57 [23:44:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.27, 10.28, 10.30 [23:46:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 4.06, 3.64 [23:46:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.74, 1.79 [23:48:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.93, 3.74 [23:48:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.89, 3.63 [23:48:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [23:49:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.58, 3.74 [23:50:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.36, 9.82, 10.18 [23:50:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.76, 3.59 [23:51:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.95, 7.29, 6.31 [23:51:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.59, 1.99, 1.46 [23:52:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.21, 3.41 [23:53:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.53, 3.64 [23:53:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.20, 1.67, 1.41 [23:53:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 11.14, 11.11 [23:54:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.65, 3.10, 3.35 [23:55:32] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.59, 7.36, 6.51 [23:55:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.14, 3.04, 3.45 [23:55:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 11.09, 11.10 [23:57:28] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 8.00, 7.48, 6.65 [23:57:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.97, 4.23, 3.84 [23:58:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.46, 3.51 [23:59:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.93, 3.78