[00:00:13] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 6.265 second response time [00:00:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.10, 11.21, 10.94 [00:00:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.92, 3.64 [00:01:10] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32404 bytes in 2.637 second response time [00:01:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 11.42, 11.27 [00:01:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.43, 11.92, 11.13 [00:02:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.49, 11.29, 10.39 [00:02:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.93, 3.67 [00:03:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.45, 1.28 [00:03:37] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 5.44, 5.21, 3.72 [00:04:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.94, 10.76, 10.31 [00:05:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 4.01, 3.71 [00:05:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.23, 1.32, 1.25 [00:06:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.01, 4.83, 3.57 [00:06:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.29, 10.99, 10.42 [00:08:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.49, 9.25, 10.08 [00:08:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.76, 10.78, 10.41 [00:09:37] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 2.55, 4.40, 3.92 [00:11:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.69, 3.47 [00:12:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.97, 11.54, 10.84 [00:13:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.54, 9.17, 10.15 [00:14:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.16, 11.50, 10.92 [00:15:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.94, 3.60 [00:16:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.29, 11.57, 11.00 [00:17:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.85, 3.99 [00:17:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.31, 10.18, 10.38 [00:18:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.56, 10.94, 10.83 [00:18:53] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 2495s [00:19:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.11, 11.34, 10.78 [00:19:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.73, 3.58 [00:21:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 4.01, 4.02 [00:21:32] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 2649s [00:21:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:21:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 12.00, 11.73, 11.01 [00:21:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.85, 3.63 [00:23:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 10.54, 10.71 [00:23:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.77, 3.64 [00:25:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 10.97, 10.86 [00:27:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.46, 11.06 [00:27:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.78, 3.64 [00:28:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.91, 9.59, 10.19 [00:29:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 11.04, 10.95 [00:29:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.26, 3.46 [00:31:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.82, 2.81, 3.27 [00:32:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.82, 11.30, 10.75 [00:33:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.46, 3.93 [00:34:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 10.76, 10.64 [00:35:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.69, 3.95 [00:37:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.51, 3.84 [00:37:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.94, 11.04, 10.49 [00:38:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.91, 11.43, 10.91 [00:39:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.70, 3.86 [00:39:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.58, 10.64, 10.40 [00:39:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.26, 8.83, 9.93 [00:39:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.14, 1.57 [00:40:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.36, 3.95 [00:41:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.57, 11.47, 10.71 [00:41:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.47, 3.77 [00:41:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.64, 1.45 [00:42:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.32, 10.97, 10.87 [00:43:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.53, 3.73 [00:44:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.72, 11.52, 10.91 [00:45:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 8.78, 9.86 [00:45:54] PROBLEM - www.cgradegames.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.cgradegames.net All nameservers failed to answer the query. [00:47:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.64, 3.74 [00:48:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.08, 3.51 [00:51:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.71, 3.96, 3.78 [00:52:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.61, 3.63 [00:52:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.69, 9.66, 10.17 [00:52:45] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 2.00, 1.86, 1.59 [00:53:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.49, 4.30, 3.57 [00:54:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.55, 1.50 [00:55:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.97, 3.83 [00:56:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.93, 3.77 [00:56:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.67, 3.48 [00:58:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.54, 3.66 [00:59:34] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [01:07:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.68, 2.77, 3.35 [01:07:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.47, 10.04 [01:09:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 10.61, 9.90 [01:10:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.71, 11.92, 11.01 [01:10:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.19, 10.46, 9.92 [01:11:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.56, 10.83, 10.32 [01:11:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.34, 3.47 [01:12:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.39, 4.73, 3.95 [01:12:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.51, 4.55, 3.95 [01:12:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.07, 9.79, 9.73 [01:13:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.31, 4.13, 3.74 [01:14:48] PROBLEM - www.cgradegames.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cgradegames.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:16:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 11.76, 11.28 [01:18:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.25, 3.57, 3.77 [01:19:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.81, 10.12, 10.05 [01:20:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.51, 11.47, 10.39 [01:20:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.88, 9.73, 10.12 [01:21:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.81, 3.87 [01:22:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.33, 11.24, 10.43 [01:22:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.25, 11.21, 11.13 [01:22:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.03, 5.03, 4.21 [01:24:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/09a1bd8038d2...ec4b33d392fd [01:24:00] [url] Comparing 09a1bd8038d2...ec4b33d392fd · miraheze/mediawiki · GitHub | github.com [01:24:01] [02miraheze/mediawiki] 07Universal-Omega 03ec4b33d - Update Cosmos [01:24:03] !log [universalomega@test101] starting deploy of {'world': True} to all [01:24:06] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [01:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 2.77, 1.96 [01:24:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.32, 11.06, 11.11 [01:25:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.51, 10.73, 10.34 [01:26:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.96, 3.98 [01:28:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.84, 4.62, 4.21 [01:29:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.75, 3.97 [01:29:32] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 329s [01:29:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.33, 11.46, 10.73 [01:30:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.76, 1.76 [01:31:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.05, 10.64, 10.53 [01:32:22] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 495s [01:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.26, 10.15, 10.02 [01:33:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 3.75, 3.89 [01:33:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.30, 9.63, 10.16 [01:34:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.62, 1.61, 1.70 [01:34:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.72, 9.75, 9.88 [01:35:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.43, 3.77 [01:35:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.99, 11.56, 10.73 [01:35:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 4.21, 3.93 [01:37:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.16, 4.01 [01:39:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.82, 3.91 [01:39:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.93, 11.56, 10.92 [01:39:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.88, 3.89 [01:41:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.93, 11.14 [01:43:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.64, 3.80 [01:43:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 11.10, 10.95 [01:43:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.74, 1.50 [01:43:54] RECOVERY - www.cgradegames.net - reverse DNS on sslhost is OK: SSL OK - www.cgradegames.net reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:46:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.51, 10.85, 10.58 [01:47:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 2.23, 1.77 [01:48:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.60, 10.17, 10.38 [01:48:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.59, 3.93 [01:49:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.77, 3.88 [01:49:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.93, 1.76, 1.66 [01:49:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.39, 10.22, 10.09 [01:50:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.78, 3.96 [01:51:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.98, 3.94 [01:51:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.97, 1.55, 1.60 [01:51:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.99, 9.73, 9.93 [01:53:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.63, 3.62 [01:55:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.33, 3.51 [01:56:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.36, 3.51, 3.85 [01:57:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.70, 3.86 [01:57:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.48, 9.69, 10.19 [01:57:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.28, 1.63, 1.94 [02:00:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 9.33, 10.01 [02:01:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.78, 3.82 [02:01:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 10.05, 10.30 [02:01:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 2.95, 3.32 [02:03:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.65, 3.76 [02:03:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.67, 9.22, 9.97 [02:08:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.82, 3.74 [02:10:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.42, 3.59 [02:10:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.61, 3.52 [02:12:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 3.21, 3.39 [02:13:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 2.88, 3.35 [02:14:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.46, 3.54 [02:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.28, 3.46 [02:17:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.40, 3.48 [02:17:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.80, 1.77 [02:18:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.09, 3.37 [02:21:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 3.89, 3.60 [02:21:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.69, 9.90, 9.37 [02:22:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.90, 3.69 [02:23:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.55, 3.51 [02:23:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [02:23:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.91, 1.85 [02:23:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.43, 9.72, 9.37 [02:23:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.80, 9.70, 9.36 [02:25:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.61, 3.52 [02:25:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.39, 8.83, 9.08 [02:26:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.79, 4.74, 4.01 [02:27:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.66, 3.55 [02:27:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 4.29, 3.68 [02:29:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.86, 3.62 [02:29:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.31, 10.67, 9.85 [02:29:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.92, 1.85 [02:29:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.43, 10.25, 9.65 [02:29:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.83, 3.59 [02:31:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.55, 10.25, 9.76 [02:31:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.18, 9.80, 9.56 [02:32:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.75, 10.00, 9.63 [02:33:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.57, 3.56 [02:33:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 10.87, 10.04 [02:34:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.84, 9.88, 9.63 [02:35:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.66, 3.58 [02:35:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.71, 10.27, 9.84 [02:35:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 3.20, 3.39 [02:37:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.56, 3.55 [02:37:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 2.00, 1.95 [02:39:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.26, 11.38, 10.37 [02:40:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 10.59, 9.94 [02:41:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.57, 1.33 [02:41:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 11.63, 10.92 [02:41:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.61, 10.41, 10.15 [02:43:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.68, 1.40 [02:43:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.94, 9.79, 9.96 [02:43:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.96, 1.93 [02:44:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.69, 3.92 [02:46:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.69, 1.87, 1.56 [02:46:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.91, 11.94, 10.80 [02:46:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.88, 3.95 [02:47:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.48, 1.59, 1.42 [02:47:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.17, 3.80, 3.64 [02:47:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.58, 9.95, 10.08 [02:48:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.55, 1.70, 1.53 [02:49:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 3.99, 3.51 [02:51:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.70, 11.20, 10.52 [02:51:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.76, 3.67 [02:51:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:51:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.64, 3.43 [02:52:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.76, 1.52 [02:53:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.31, 10.77, 10.38 [02:53:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 4.19, 3.67 [02:54:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.43, 11.12, 10.62 [02:54:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 6.00, 3.41, 2.15 [02:55:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.34, 11.60, 10.72 [02:56:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [02:56:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.38, 11.67, 11.19 [02:56:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.92, 11.65, 10.86 [02:58:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.99, 11.09, 11.05 [02:58:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.50, 3.86 [02:58:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.47, 10.56, 10.56 [02:59:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.53, 2.76, 3.35 [02:59:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.62, 3.64 [03:01:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.41, 11.65, 11.08 [03:01:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.39, 2.85, 3.35 [03:02:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.33, 11.96, 11.39 [03:04:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.87, 11.69, 11.38 [03:04:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 2.68, 3.36 [03:05:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.20, 4.01, 3.68 [03:06:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 3.73, 3.55 [03:06:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.12, 11.71, 11.41 [03:06:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.94, 1.69, 1.92 [03:07:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.76, 9.06, 10.10 [03:07:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.69, 3.60 [03:07:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.31, 3.42 [03:08:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 10.87, 11.14 [03:08:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.05, 3.87, 3.66 [03:09:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.33, 11.09 [03:09:35] [discord] Any SREs around to assist in turning on SMW? [03:10:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.04, 11.33, 11.27 [03:11:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.68, 10.67, 10.88 [03:11:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 4.20, 3.79 [03:11:53] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.91, 1.60 [03:11:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.91, 4.02, 3.76 [03:12:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.28, 1.66 [03:13:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.60, 3.62 [03:13:52] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.74, 1.58 [03:13:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.63, 3.64 [03:14:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.73, 10.78, 11.13 [03:15:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 4.46, 3.97 [03:15:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.12, 1.57, 1.54 [03:16:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.95, 3.82 [03:17:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.92, 3.83 [03:20:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 4.06, 3.88 [03:21:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.49, 4.08 [03:21:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.58, 9.94 [03:21:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.83, 3.69 [03:22:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.77, 1.82, 1.66 [03:22:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.24, 9.45, 10.06 [03:22:46] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.33, 1.74, 1.64 [03:24:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.83, 1.68 [03:24:45] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.27, 1.64, 1.62 [03:25:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.70, 3.67 [03:26:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.29, 2.00, 1.76 [03:27:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.73, 11.15 [03:27:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.92, 3.97 [03:27:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.10, 11.25, 10.40 [03:28:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.86, 1.74 [03:29:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.07, 11.40, 10.58 [03:31:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.27, 4.20, 4.02 [03:31:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.42, 11.32, 11.14 [03:31:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.76, 2.00 [03:32:29] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.59, 1.65 [03:32:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.91, 10.90, 10.31 [03:33:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.90, 3.93 [03:35:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.01, 9.77, 10.08 [03:36:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.31, 3.83 [03:36:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.49, 11.01, 10.43 [03:37:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 4.00, 3.96 [03:38:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.96, 11.06, 10.53 [03:39:37] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.39, 1.70 [03:39:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.70, 4.19, 3.80 [03:40:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.66, 10.84, 10.60 [03:40:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 4.27, 4.11 [03:42:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.06, 10.97, 10.68 [03:44:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.47, 9.61, 10.08 [03:46:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.14, 9.76, 10.20 [03:47:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.77, 1.55 [03:48:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.60, 10.16, 10.22 [03:49:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.45, 1.67, 1.54 [03:49:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.88, 2.05, 1.79 [03:50:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.27, 10.63, 10.52 [03:50:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.61, 9.97, 10.16 [03:51:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.57, 9.66, 10.13 [03:52:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 9.38, 10.07 [03:53:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.82, 10.31, 9.98 [03:55:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.97, 1.88 [03:56:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:59:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.63, 10.09, 10.01 [04:02:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 10.37, 9.99 [04:03:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 10.49, 10.22 [04:04:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.16, 9.98, 9.91 [04:05:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.06, 10.35, 10.12 [04:05:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.15, 9.96, 10.08 [04:07:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.20, 9.87, 9.97 [04:09:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.74, 10.23, 10.13 [04:10:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.85, 11.46, 10.47 [04:11:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 9.52, 9.88 [04:11:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.91, 1.83 [04:18:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.17, 11.80, 11.16 [04:26:34] PROBLEM - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.410584449768066 seconds [04:27:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 1.44, 1.20 [04:27:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.91, 1.49 [04:29:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.50, 1.46, 1.23 [04:29:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.56, 1.41 [04:32:16] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.43, 10.26, 9.88 [04:34:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.45, 9.73, 9.75 [04:36:26] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.56, 6.26, 3.78 [04:36:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.31, 9.10, 10.03 [04:38:28] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=fictionwiki --u=Miraheze_maintenance_script --r=[[phab:T9492 [04:38:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:35] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=fictionwiki --u=Miraheze_maintenance_script --r=[[phab:T9492 [04:38:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:53] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/90661e92915a...2a22c92e6d12 [04:42:54] [url] Comparing 90661e92915a...2a22c92e6d12 · miraheze/ssl · GitHub | github.com [04:42:55] [02miraheze/ssl] 07Reception123 032a22c92 - add lab.wikimimo.com cert [04:45:37] [discord] Reception123: Can you handle the SMW request on the SN? [04:46:14] sure [04:46:26] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.85, 7.89, 6.22 [04:48:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 9.49, 9.17 [04:48:26] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.08, 6.23, 5.81 [04:50:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 10.55, 9.60 [04:50:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.59, 9.96, 9.62 [04:50:37] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [04:51:19] !log [salt-user@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=65280) [04:51:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.55, 10.70, 9.89 [04:51:37] !log [salt-user@mw101] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [04:51:40] !log [salt-user@mw102] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [04:51:41] !log [salt-user@mw111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [04:51:42] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [04:51:44] !log [salt-user@mw112] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [04:51:45] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [04:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:51:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.44, 9.85, 9.44 [04:52:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.31, 9.57, 9.51 [04:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:55:07] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [04:55:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.37, 9.81, 9.73 [04:55:23] RECOVERY - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is OK: SSL OK - wiki.rosestulipsandliberty.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:55:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 10.71, 9.91 [04:57:26] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [04:57:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.04, 10.38, 9.51 [04:57:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.05, 10.11 [04:57:49] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:00:33] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [05:02:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.37, 9.86, 9.53 [05:02:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.86, 10.29 [05:04:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 11.33, 10.51 [05:04:41] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:05:54] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:06:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.61, 10.80, 9.97 [05:07:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 10.06, 9.64 [05:07:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.49, 3.98 [05:07:37] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:08:09] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.71, 1.93 [05:09:41] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:09:50] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:10:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.04, 11.44, 10.91 [05:10:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.40, 11.58, 10.31 [05:12:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.66, 11.63, 10.49 [05:13:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.94, 10.01, 9.95 [05:13:44] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:13:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.31, 3.89 [05:14:21] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:15:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.77, 3.98 [05:17:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.61, 3.91 [05:18:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 1.86, 1.84 [05:18:25] !log DELETED and DROPPED turkisiwiki (reset) [05:18:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.64, 3.96 [05:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:19:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.72, 3.90 [05:19:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.35, 9.28, 10.05 [05:20:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 4.04, 4.07 [05:21:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.60, 3.84 [05:22:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.42, 10.33, 10.00 [05:22:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.59, 9.97, 10.11 [05:23:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.77, 3.39, 1.94 [05:23:23] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=aralghowiki (END - exit=0) [05:23:28] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:42] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:24:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 10.70, 10.16 [05:24:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.43, 3.31, 3.79 [05:25:33] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [05:25:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.79, 3.86 [05:25:52] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:25:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.28, 9.74, 9.87 [05:25:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.91, 1.93 [05:26:52] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/lab.wikimimo.com.key] [05:27:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.75, 1.95, 1.67 [05:27:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.36, 3.69 [05:29:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.41, 1.40, 1.50 [05:30:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.79, 3.06, 3.35 [05:31:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.24, 10.09, 10.09 [05:33:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 9.08, 9.72 [05:33:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 2.81, 3.31 [05:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.76, 10.88, 10.15 [05:35:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 2.92, 3.36 [05:35:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.26, 10.03, 9.91 [05:41:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.28, 3.24 [05:41:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.52, 3.46 [05:41:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.84, 1.83 [05:43:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 2.87, 3.09 [05:43:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.80, 10.37, 10.02 [05:43:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.72, 3.53 [05:43:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.81, 1.82 [05:45:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.79, 3.33, 3.40 [05:47:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.99, 9.48, 9.77 [05:47:46] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.95, 2.12, 1.92 [05:55:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 2.00, 1.98 [05:57:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.99, 2.07, 2.00 [05:59:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.33, 3.34 [06:05:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.78, 3.54 [06:05:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.36, 10.77, 9.29 [06:07:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.81, 3.12, 3.33 [06:07:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.19, 10.91, 9.53 [06:11:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.89, 10.10, 9.53 [06:12:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.91, 10.95, 9.79 [06:13:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.60, 3.48 [06:15:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.13, 3.31 [06:16:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.89, 9.98, 9.67 [06:17:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.43, 10.39, 9.81 [06:23:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 10.08, 9.93 [06:25:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.51, 3.34 [06:26:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.31, 3.13 [06:28:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 4.04, 3.41 [06:29:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.81, 10.37, 10.06 [06:29:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.81, 3.51 [06:30:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.85, 3.42 [06:31:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.60, 11.24, 10.41 [06:31:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.32, 3.28, 3.35 [06:32:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 3.16, 3.21 [06:33:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.13, 10.72, 10.33 [06:34:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.85, 3.42 [06:36:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.25, 3.61 [06:37:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.74, 10.18, 10.19 [06:38:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.51, 3.46 [06:40:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.52, 3.36 [06:40:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.68, 3.53 [06:42:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.37, 3.21, 3.34 [06:42:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.70, 10.56, 10.04 [06:43:52] paladox: what command did we use for deleting /thumb again for https://phabricator.miraheze.org/T8346 ? [06:43:52] [url] ⚓ T8346 Consider a one-off deletion of /thumb accross all wikis | phabricator.miraheze.org [06:43:55] I'd like to try again [06:44:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 3.15, 3.28 [06:44:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.87, 10.02, 9.91 [06:44:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.14, 3.28, 3.39 [06:44:32] also [06:44:33] Warning: The directory '/home/ssl-admins' contains 6067 entries, which exceeds the default soft limit 1000 and may cause excessive resource consumption and degraded performance. To remove this warning set a value for `max_files` parameter or consider using an alternate method to manage large directory trees [06:45:05] puppet is spamming things like [06:45:06] Notice: /Stage[main]/Ssl/File[/home/ssl-admins/ssl-keys/.git/objects/ff/8cc31206763ab9f7168a9f53db1f35072446f9]/mode: mode changed '0660' to '0770' (corrective) [06:48:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 10.82, 10.16 [06:49:02] Reception123: deleting /thumb is pointless [06:49:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.82, 10.59, 9.81 [06:50:41] RhinosF1: it might seem like it but there's many dead wikis/dead files out there so it does something even if minor I think [06:50:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 11.38, 10.46 [06:50:52] Reception123: also, not sure what's best out there [06:51:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.04, 10.75, 9.94 [06:51:07] any idea what's going on with the puppet issue though? [06:51:14] Reception123: it's unlikely to have any meaningful benefit [06:51:21] Reception123: I know exactly what caused it [06:51:33] I mean I know it must be MacFan's PR but I thought it worked before? [06:51:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.79, 3.51 [06:51:38] I just have no idea what's best to do [06:51:42] ah [06:51:43] Yeah it is [06:52:29] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 6.22, 5.42, 4.05 [06:52:48] It's likely cause the git repo [06:52:59] Being too deep [06:53:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.96, 9.95, 9.77 [06:53:33] I don't really want to just say up max_files [06:54:29] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 3.42, 4.62, 3.92 [06:54:53] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:55:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.17, 3.27, 3.38 [06:57:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.83, 10.97, 10.20 [06:57:23] RhinosF1: I mean we've done fine without forcing perms before so we could just revert [06:57:30] issues with perms have been quite rare [06:57:49] Reception123: we can turn off recurse here [06:57:50] Ye [06:57:59] You only need to revert one line [06:58:13] yeah I'd revert the recurse [06:58:29] Feel free too [06:59:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.25, 11.13, 10.36 [06:59:13] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9e9b530b340c...3303fb1b563c [06:59:13] [url] Comparing 9e9b530b340c...3303fb1b563c · miraheze/puppet · GitHub | github.com [06:59:14] [02miraheze/puppet] 07Reception123 033303fb1 - don't recurse ssl-keys [07:01:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.71, 10.15, 10.09 [07:02:29] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 6.08, 5.67, 4.63 [07:04:29] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 2.49, 4.39, 4.28 [07:09:27] Reception123: the 2nd one was working fine [07:10:08] Well it was also fine without it before wasn't it? [07:10:39] Reception123: well apparently they were permission issues [07:10:49] Obviously enough MacFan4000 cared [07:11:31] Well we can add it back if we're sure it works but it does spam puppet quite a bit and I'm pretty sure the same warning would apply [07:13:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.88, 10.64, 9.91 [07:16:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.37, 10.23, 9.75 [07:17:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.49, 9.44, 9.63 [07:17:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.12, 2.97 [07:17:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.02, 9.65, 9.60 [07:18:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 9.29, 10.07 [07:19:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 2.72, 2.84 [07:20:53] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:32] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.82, 1.39 [07:29:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.49, 1.32 [07:42:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 10.88, 9.80 [07:44:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.08, 10.33, 9.72 [07:46:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 10.88, 9.97 [07:47:54] 1:10 AM Obviously enough MacFan4000 cared — what was the issue there? It always worked for me anyway. [07:48:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.11, 10.24, 9.85 [07:48:25] CosmicAlpha: a few files had permissions meaning ssl-admins couldn't read [07:49:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.00, 2.73 [07:49:18] RhinosF1: Every one of them was owned by MacFan4000, but it still worked for me. [07:49:36] CosmicAlpha: ok [07:50:05] Anyway I'm off to sleep. [07:50:17] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.25, 9.69, 9.68 [07:50:26] Night [07:51:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 2.95, 2.74 [07:55:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 4.17, 3.36 [07:57:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.07, 5.32, 3.80 [07:58:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.85, 3.07 [07:59:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.00, 10.23 [07:59:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.82, 3.40 [08:00:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.52, 3.04 [08:00:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.58, 10.22, 9.39 [08:00:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.27, 10.73, 10.22 [08:01:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dcbac7ef0989...c6888a3f5c6a [08:01:16] [url] Comparing dcbac7ef0989...c6888a3f5c6a · miraheze/mw-config · GitHub | github.com [08:01:17] [02miraheze/mw-config] 07Universal-Omega 03c6888a3 - Set wgVectorDefaultSkinVersion to '1' [08:02:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.21, 2.98 [08:02:37] miraheze/mw-config - Universal-Omega the build passed. [08:02:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.56, 9.67, 9.29 [08:05:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.35, 3.24, 3.26 [08:08:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.05, 10.90, 10.46 [08:08:53] FYI, I am away until Wednsday now. [08:09:18] Reception123, RhinosF1: ^ [08:09:58] Will be out of town with no PC access at all. [08:10:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.75, 11.14, 10.60 [08:11:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 10.65, 9.92 [08:12:29] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.29, 11.62, 10.84 [08:14:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.29, 11.17, 10.77 [08:14:27] ok CosmicAlpha|AWAY [08:15:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.39, 9.70, 9.72 [08:18:36] Ack [08:22:38] !log [@mwtask111] starting deploy of {'config': True} to all [08:22:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:22:53] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 14s [08:22:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.38, 8.54, 9.84 [08:25:05] !log [@test101] starting deploy of {'config': True} to all [08:25:07] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [08:25:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:36:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 10.60, 9.79 [08:38:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.78, 10.16, 9.73 [08:41:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.01, 10.35, 9.93 [08:42:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.66, 10.84, 10.17 [08:43:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.80, 9.73, 9.75 [08:44:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.77, 9.88, 9.89 [09:04:57] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.67, 1.76, 1.39 [09:06:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 10.02, 9.55 [09:06:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 10.18, 9.34 [09:06:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 1.76, 1.43 [09:08:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.66, 9.12, 9.05 [09:09:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.56, 9.81, 9.60 [09:10:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.31, 1.62, 1.44 [09:12:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.43, 10.16, 9.33 [09:14:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.33, 9.59, 9.23 [09:18:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.58, 10.70, 9.75 [09:21:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.92, 9.96, 9.67 [09:22:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.52, 3.14 [09:24:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.83, 3.09, 3.04 [09:41:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.46, 10.69, 9.94 [09:49:20] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.48, 11.20, 10.38 [09:51:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.04, 10.86, 10.37 [09:55:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.85, 9.92, 10.11 [10:05:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.34, 1.04 [10:07:03] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.41, 1.31, 1.06 [10:11:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.33, 10.05, 9.94 [10:13:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.78, 9.62, 9.79 [10:30:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.65, 2.00 [10:31:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.09, 10.51, 9.58 [10:31:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.97, 9.29, 5.91 [10:33:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.71, 9.99, 9.49 [10:38:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.34, 1.70 [10:42:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 10.42, 9.92 [10:43:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.65, 10.05, 9.84 [10:47:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.05, 2.49, 1.71 [10:47:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 2.00, 1.85 [10:48:07] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14251 MB (10% inode=99%); [10:49:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.19, 2.00, 1.62 [10:49:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.42, 5.28, 5.88 [10:51:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.90, 2.93, 3.96 [10:51:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.98, 1.63, 1.53 [10:55:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 18.56, 7.55, 6.26 [10:57:20] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.91, 1.98 [10:59:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.99, 5.09, 5.54 [11:01:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.91, 1.95 [11:03:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 2.77, 3.36 [11:03:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.77, 5.58, 5.62 [11:04:07] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 47286 MB (35% inode=99%); [11:05:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.77, 1.88 [11:05:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.92, 4.71, 5.31 [11:07:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.24, 8.07, 6.54 [11:07:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.42, 3.02 [11:09:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.30, 3.31, 3.03 [11:10:33] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1d4b14bed522...5c092d0a0faf [11:10:34] [url] Comparing 1d4b14bed522...5c092d0a0faf · miraheze/ImportDump · GitHub | github.com [11:10:35] [02miraheze/ImportDump] 07translatewiki 035c092d0 - Localisation updates from https://translatewiki.net. [11:10:35] [url] Main page - translatewiki.net | translatewiki.net [11:10:36] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e96869bb3c1d...7fe5ac9fcd49 [11:10:37] [url] Comparing e96869bb3c1d...7fe5ac9fcd49 · miraheze/CreateWiki · GitHub | github.com [11:10:38] [02miraheze/CreateWiki] 07translatewiki 037fe5ac9 - Localisation updates from https://translatewiki.net. [11:10:38] [url] Main page - translatewiki.net | translatewiki.net [11:10:39] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/e4019e767a99...823411dd3171 [11:10:40] [url] Comparing e4019e767a99...823411dd3171 · miraheze/landing · GitHub | github.com [11:10:41] [02miraheze/landing] 07translatewiki 03823411d - Localisation updates from https://translatewiki.net. [11:10:41] [url] Main page - translatewiki.net | translatewiki.net [11:10:42] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/d992b714842d...311a0068fb69 [11:10:43] [url] Comparing d992b714842d...311a0068fb69 · miraheze/MirahezeMagic · GitHub | github.com [11:10:44] [02miraheze/MirahezeMagic] 07translatewiki 03311a006 - Localisation updates from https://translatewiki.net. [11:10:44] [url] Main page - translatewiki.net | translatewiki.net [11:12:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.96, 10.79, 9.59 [11:13:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 2.08, 1.98 [11:13:29] miraheze/landing - translatewiki the build passed. [11:16:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.82, 9.85, 9.44 [11:18:37] miraheze/ImportDump - translatewiki the build passed. [11:20:47] miraheze/CreateWiki - translatewiki the build passed. [11:21:54] miraheze/MirahezeMagic - translatewiki the build passed. [11:28:30] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 6.62, 5.00, 3.25 [11:30:29] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 4.67, 4.98, 3.47 [11:36:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.95, 9.91, 9.11 [11:38:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.99, 9.52, 9.07 [11:41:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.95, 3.73, 5.95 [11:43:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.25, 4.23, 3.34 [11:44:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.84, 3.21 [11:45:30] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.85, 2.23, 4.83 [11:45:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.60, 3.10 [11:47:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.91, 3.45 [11:48:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.37, 3.15 [11:51:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.46, 9.50, 8.80 [11:51:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.00, 2.88, 2.97 [11:53:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.74, 9.28, 8.82 [11:53:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 3.03, 3.24 [11:58:43] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.79, 1.99, 1.51 [12:02:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.90, 1.49, 1.41 [12:03:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.34, 3.06 [12:05:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.05, 3.25, 3.06 [12:08:41] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.39, 1.71, 1.97 [12:10:40] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.85, 1.99 [12:12:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.68, 3.29 [12:12:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.28, 1.61, 1.88 [12:14:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.83, 2.18, 1.73 [12:14:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 1.85, 1.93 [12:16:05] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.98, 1.70 [12:16:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.70, 1.86 [12:17:01] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:18:00] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.88, 1.59, 1.59 [12:18:36] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 1.99, 1.95 [12:26:13] PROBLEM - orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:26:32] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.39, 1.78, 1.94 [12:26:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.09, 1.78, 1.64 [12:26:42] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.99, 3.14, 2.58 [12:28:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.00, 10.54, 10.02 [12:28:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.47, 1.55 [12:28:41] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.08, 3.20, 2.68 [12:29:10] The problem was that only root could touch them [12:29:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 4.32, 3.56 [12:30:30] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 2.02, 1.99 [12:30:41] Without recourse it will only ensure permissions for the base directory. [12:31:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.77, 11.01, 10.06 [12:32:11] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.42, 10.13, 9.97 [12:32:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.91, 1.95 [12:33:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.82, 11.10, 10.23 [12:33:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.65, 3.47 [12:35:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.17, 3.32 [12:36:20] PROBLEM - miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.wiki' expires in 15 day(s) (Wed 27 Jul 2022 12:15:15 GMT +0000). [12:37:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.54, 11.63, 10.61 [12:37:34] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:38:27] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [12:39:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 10.73, 10.41 [12:42:07] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.80, 1.57 [12:42:29] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.84, 1.87 [12:42:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.65, 10.49, 10.04 [12:43:37] Reception123: ^^ wildcards that need renewal [12:44:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.37, 1.60, 1.52 [12:44:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.89, 3.60 [12:44:44] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.15, 9.95, 9.90 [12:45:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.44, 9.44, 9.98 [12:46:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.89, 1.88 [12:48:29] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.61, 2.16, 1.98 [12:51:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.45, 4.09, 3.70 [12:53:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.61, 3.56 [12:55:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.96, 4.14, 3.79 [12:56:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.69, 3.37 [12:57:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.84, 3.72 [13:00:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.59, 3.37 [13:02:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 3.02, 3.18 [13:03:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.38, 10.53, 9.89 [13:05:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.97, 3.08, 3.36 [13:07:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.35, 9.51, 9.64 [13:16:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.95, 10.74, 9.99 [13:18:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.59, 3.39 [13:18:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.06, 11.00, 10.16 [13:20:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.07, 3.38, 3.34 [13:20:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.27, 11.13, 10.31 [13:23:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.90, 3.59 [13:28:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.07, 9.36, 9.85 [13:31:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 2.83, 3.26 [13:37:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.52, 3.46 [13:38:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.37, 3.16 [13:40:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.32, 3.16, 3.12 [13:41:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.13, 3.20, 3.39 [13:54:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.43, 3.36 [13:56:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.08, 3.23 [14:00:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.56, 3.35 [14:00:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.72, 1.98 [14:02:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 3.15, 3.23 [14:06:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.68, 3.43 [14:08:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.37, 3.35 [14:08:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.34, 1.68 [14:11:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.55, 3.31 [14:12:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.50, 3.42 [14:13:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.25, 10.66, 9.85 [14:14:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.78, 3.53 [14:14:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 4.11, 3.58 [14:15:16] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.63, 3.32, 2.92 [14:15:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.42, 9.82, 9.64 [14:16:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.77, 3.56 [14:16:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.27, 4.38, 2.35 [14:16:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.73, 1.73 [14:18:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.23, 10.23, 9.66 [14:18:29] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 2.08, 1.86 [14:19:13] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.70, 3.16, 2.96 [14:20:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.16, 10.14, 9.69 [14:20:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.83, 3.63 [14:22:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.47, 3.45 [14:22:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.34, 4.31, 3.82 [14:24:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.24, 3.36 [14:26:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.89, 1.91 [14:26:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.96, 3.78 [14:26:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.36, 10.64, 10.04 [14:28:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.75, 3.53 [14:28:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.15, 4.43, 3.98 [14:28:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 1.97, 1.95 [14:30:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.77, 1.36, 1.69 [14:30:29] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.98, 1.95 [14:32:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.62, 3.52 [14:32:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.89, 3.88 [14:32:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.80, 1.88 [14:32:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.06, 11.51, 10.52 [14:33:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.18, 4.27, 4.01 [14:34:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.76, 3.56 [14:34:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.44, 11.08, 10.48 [14:35:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.71, 3.83 [14:38:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.72, 3.58 [14:38:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.46, 11.96, 10.94 [14:38:29] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.98, 1.93 [14:39:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.10, 11.31, 10.02 [14:40:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.48, 11.56, 10.94 [14:42:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.75, 3.95, 3.63 [14:42:05] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/fc3f4338808d [14:42:06] [02miraheze/ssl] 07Universal-Omega 03fc3f433 - Add source.erag.eu.org cert [14:42:08] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [14:42:08] [url] Page not found · GitHub · GitHub | github.com [14:42:09] [02ssl] 07Universal-Omega opened pull request 03#564: Add source.erag.eu.org cert - 13https://github.com/miraheze/ssl/pull/564 [14:42:10] [url] Page not found · GitHub · GitHub | github.com [14:43:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.57, 3.67 [14:43:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.73, 11.54, 10.46 [14:44:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:44:03] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fc3f4338808d...abff1690bf9f [14:44:04] [url] Comparing fc3f4338808d...abff1690bf9f · miraheze/ssl · GitHub | github.com [14:44:05] [02miraheze/ssl] 07Universal-Omega 03abff169 - Update certs.yaml [14:44:06] [02ssl] 07Universal-Omega synchronize pull request 03#564: Add source.erag.eu.org cert - 13https://github.com/miraheze/ssl/pull/564 [14:44:07] [url] Page not found · GitHub · GitHub | github.com [14:45:54] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/abff1690bf9f...54e5fdb7eba1 [14:45:54] [url] Comparing abff1690bf9f...54e5fdb7eba1 · miraheze/ssl · GitHub | github.com [14:45:55] [02miraheze/ssl] 07Universal-Omega 0354e5fdb - Update redirects.yaml [14:45:57] [02ssl] 07Universal-Omega synchronize pull request 03#564: Add source.erag.eu.org cert - 13https://github.com/miraheze/ssl/pull/564 [14:45:57] [url] Page not found · GitHub · GitHub | github.com [14:46:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.68, 3.64 [14:47:17] [02ssl] 07Universal-Omega edited pull request 03#564: Add source.erag.eu.org cert - 13https://github.com/miraheze/ssl/pull/564 [14:47:18] [url] Page not found · GitHub · GitHub | github.com [14:47:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.56, 3.69 [14:48:56] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [14:48:57] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/ssl/compare/2a22c92e6d12...81d6b2cedfa5 [14:48:58] [url] Comparing 2a22c92e6d12...81d6b2cedfa5 · miraheze/ssl · GitHub | github.com [14:48:59] [02miraheze/ssl] 07Universal-Omega 0381d6b2c - Add source.erag.eu.org cert [14:49:00] [02ssl] 07Universal-Omega closed pull request 03#564: Add source.erag.eu.org cert - 13https://github.com/miraheze/ssl/pull/564 [14:49:00] ... [14:49:02] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [14:49:02] [url] Page not found · GitHub · GitHub | github.com [14:50:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 3.02, 3.38 [14:50:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.78, 1.83, 1.34 [14:51:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.62, 3.69, 3.65 [14:51:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.38, 9.20, 9.98 [14:52:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.97, 1.46 [14:53:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.34, 3.52 [14:54:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.67, 1.40 [14:55:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 3.53, 3.55 [14:57:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.20, 3.43 [14:59:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:01:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.42, 3.41 [15:02:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 3.18, 3.37 [15:03:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.22, 3.34 [15:04:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.39, 10.52, 10.27 [15:05:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 10.08, 8.79 [15:06:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.03, 9.92, 10.07 [15:06:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.49, 3.46 [15:07:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.46, 9.80, 8.85 [15:08:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.36, 3.42 [15:10:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.30, 11.49, 11.00 [15:14:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.02, 3.26, 3.39 [15:14:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.39, 10.93, 10.95 [15:18:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.72, 3.59 [15:18:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.59, 3.45 [15:21:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.75, 10.36, 9.94 [15:22:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.91, 2.87, 3.29 [15:22:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.63, 11.21, 10.31 [15:24:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.28, 3.30, 3.37 [15:26:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.63, 11.53, 10.64 [15:26:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.95, 11.16, 10.86 [15:28:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 4.26, 3.71 [15:28:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.77, 10.49, 10.64 [15:29:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 4.86, 4.00 [15:30:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 4.00, 3.68 [15:32:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.21, 4.37, 3.85 [15:32:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.65, 11.60, 10.85 [15:34:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.43, 10.81, 10.68 [15:36:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.15, 3.63, 3.71 [15:40:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 11.94, 11.41 [15:40:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.18, 4.45, 3.99 [15:42:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.46, 12.57, 11.70 [15:43:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.67, 1.93 [15:43:51] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:44:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.80, 3.95 [15:45:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.75, 1.92 [15:45:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.38, 12.00, 11.67 [15:46:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.83, 3.87 [15:46:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 4.01, 3.99 [15:48:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 11.15, 11.32 [15:49:27] PROBLEM - fortis.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['curitiba.porkbun.com.', 'maceio.porkbun.com.', 'fortaleza.porkbun.com.', 'salvador.porkbun.com.'], 'CNAME': None} [15:50:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 3.98, 3.86 [15:51:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.59, 11.83, 11.66 [15:52:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.65, 3.77 [15:52:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.41, 3.83, 3.95 [15:53:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.51, 10.69, 11.24 [15:55:37] PROBLEM - www.fortis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fortis.wiki could not be found [15:55:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.25, 11.43, 11.46 [15:55:42] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:57:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.90, 2.00 [15:57:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:57:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.31, 10.60, 11.15 [15:58:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 3.96, 3.80 [15:58:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 4.56, 4.17 [15:59:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.76, 3.74 [16:01:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 2.12, 2.06 [16:03:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.95, 2.00 [16:03:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.18, 4.42, 4.00 [16:05:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 2.09, 2.05 [16:07:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.93, 1.99 [16:08:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.81, 9.32, 10.05 [16:09:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.21, 10.80, 10.82 [16:09:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.94, 3.94 [16:11:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.92, 1.97 [16:11:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.30, 11.02, 10.90 [16:12:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.65, 10.79, 10.47 [16:13:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.87, 1.94 [16:15:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.43, 1.95, 1.95 [16:15:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.28, 11.63, 11.14 [16:16:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.40, 3.88 [16:16:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.27, 11.73, 10.91 [16:17:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.60, 11.20, 11.04 [16:18:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.32, 11.41, 10.91 [16:24:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.68, 3.75 [16:24:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 9.80, 8.63 [16:26:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.78, 3.79 [16:26:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.45, 9.27, 8.59 [16:27:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.74, 3.62 [16:28:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.40, 11.31, 11.04 [16:29:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.60, 3.59 [16:30:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.88, 3.82 [16:31:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.62, 11.20, 11.04 [16:32:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.56, 3.70 [16:32:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.63, 10.94, 11.00 [16:33:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.29, 10.50, 10.81 [16:35:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.00, 3.04, 3.38 [16:38:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.52, 3.60 [16:39:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.57, 3.55 [16:40:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.29, 3.50 [16:41:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.19, 8.53, 9.79 [16:42:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [16:43:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.76, 4.43, 3.83 [16:43:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 10.95, 9.94 [16:44:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 3.97, 3.71 [16:47:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.05, 9.62, 9.63 [16:48:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.87, 11.04, 10.77 [16:50:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.72, 3.71 [16:50:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.02, 10.77, 10.30 [16:51:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.92, 3.88 [16:52:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.75, 4.34, 3.92 [16:53:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.18, 4.21, 3.97 [16:54:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.37, 10.92, 10.90 [16:55:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.83, 3.86 [16:56:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.25, 11.81, 11.23 [16:57:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 4.20, 3.99 [16:58:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 11.60, 11.23 [17:00:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.29, 11.63, 10.89 [17:00:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 10.28, 9.91 [17:00:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.25, 11.57, 11.25 [17:02:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.69, 9.91, 9.84 [17:03:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.81, 3.91 [17:04:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 11.78, 11.41 [17:05:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.97, 3.96 [17:06:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 10.26, 10.02 [17:08:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.12, 12.05, 11.56 [17:10:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.04, 10.17, 10.04 [17:11:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.68, 3.89 [17:11:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.87, 1.45, 1.27 [17:11:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.99, 11.87, 11.84 [17:12:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:13:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 4.16, 4.04 [17:13:36] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.35, 1.40, 1.27 [17:13:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.36, 12.09, 11.92 [17:14:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.58, 10.61, 10.22 [17:15:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.74, 11.73, 11.83 [17:17:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.19, 12.04, 11.93 [17:18:14] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.03, 3.90, 2.36 [17:18:43] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.95, 2.75, 1.70 [17:18:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.93, 11.44, 11.70 [17:19:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.77, 3.93 [17:19:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.09, 11.37, 11.71 [17:20:14] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.77, 3.92, 2.57 [17:20:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.65, 12.04, 11.90 [17:21:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.09, 11.75, 11.81 [17:22:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.71, 3.96 [17:22:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.98, 11.15, 10.53 [17:22:43] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.64, 3.62, 2.30 [17:22:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.26, 11.88, 11.88 [17:23:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.91, 3.92 [17:24:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.84, 3.98 [17:24:42] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.06, 3.03, 2.25 [17:26:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 11.74, 10.94 [17:27:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.95, 11.57, 11.82 [17:28:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.85, 3.96 [17:28:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.27, 12.17, 11.18 [17:28:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.00, 12.14, 11.88 [17:30:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 11.57, 11.07 [17:30:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.82, 11.30, 11.64 [17:31:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.70, 1.99 [17:31:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 11.82, 11.85 [17:32:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.66, 4.87, 4.30 [17:32:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [17:35:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.76, 1.94 [17:35:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.73, 11.31, 11.65 [17:37:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.69, 1.88 [17:38:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 9.20, 10.07 [17:41:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.38, 11.06, 11.33 [17:43:11] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:33] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:35] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.76, 10.89, 11.20 [17:43:38] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:51] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:54] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:31] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:41] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:45] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.48, 11.72, 11.59 [17:45:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.92, 1.86 [17:45:45] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:16] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.712 second response time [17:47:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.64, 12.11, 11.59 [17:47:46] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.767 second response time [17:47:47] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.860 second response time [17:48:45] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.448 second response time [17:48:49] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.011 second response time [17:49:43] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.821 second response time [17:49:47] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 5.329 second response time [17:49:56] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.553 second response time [17:50:01] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.729 second response time [17:50:51] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.579 second response time [17:54:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.83, 10.85, 10.18 [18:00:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.40, 10.08, 10.09 [18:01:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.30, 3.86 [18:02:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [18:06:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.81, 11.72, 11.98 [18:10:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 11.69, 11.86 [18:11:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 2.81, 3.35 [18:13:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.89, 10.96, 11.73 [18:18:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 11.86, 12.00 [18:20:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.51, 12.03, 12.04 [18:21:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.16, 11.34, 11.54 [18:22:37] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 2.19, 1.70 [18:23:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.84, 10.71, 11.29 [18:25:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.45, 3.37 [18:26:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.72, 3.98 [18:26:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.44, 1.96, 1.72 [18:27:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [18:28:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.97, 4.04 [18:28:44] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.98, 3.42, 2.20 [18:29:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.26, 3.32 [18:31:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.24, 11.31, 11.18 [18:32:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.37, 11.72, 11.95 [18:33:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 4.24, 3.69 [18:34:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.56, 1.62 [18:34:40] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.84, 1.66, 1.78 [18:34:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.16, 12.14, 12.06 [18:35:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.85, 11.78, 10.59 [18:35:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.95, 3.65 [18:35:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.48, 11.55, 11.33 [18:35:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 10.50, 9.50 [18:37:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.60, 10.66 [18:37:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.91, 11.93, 11.49 [18:38:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.59, 1.25, 1.60 [18:39:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 3.92, 3.67 [18:39:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.76, 11.78, 11.48 [18:39:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.82, 10.07, 9.58 [18:45:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.88, 3.75 [18:49:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.72, 3.68 [18:49:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.58, 11.92, 11.47 [18:50:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.64, 11.77, 10.90 [18:50:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.11, 9.70, 8.22 [18:51:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.71, 3.70 [18:52:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.64, 9.31, 8.26 [18:53:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.51, 1.96, 1.67 [18:53:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.51, 11.64, 11.47 [18:54:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.46, 11.94, 11.20 [18:56:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [18:57:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.44, 1.52 [18:57:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 4.18, 3.82 [18:58:22] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.81, 2.03, 1.58 [19:00:21] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.85, 1.56, 1.46 [19:01:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 2.10, 1.77 [19:01:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.74, 12.10, 11.64 [19:03:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.65, 1.64 [19:03:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.75, 11.93, 11.66 [19:07:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.16, 11.35, 11.43 [19:13:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.77, 11.64, 11.59 [19:15:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.81, 11.78, 11.62 [19:17:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.53, 11.77, 11.65 [19:19:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.21, 12.08, 11.79 [19:20:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.34, 9.46, 10.14 [19:21:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.71, 3.95 [19:21:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.31, 11.50, 11.62 [19:23:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.77, 3.93 [19:24:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 10.11, 10.22 [19:24:56] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.45, 3.08, 2.68 [19:26:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.61, 10.86, 11.66 [19:26:56] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.43, 2.82, 2.63 [19:28:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.74, 8.92, 9.76 [19:31:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.47, 3.84 [19:31:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.17, 8.40, 10.19 [19:32:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.05, 7.22, 9.74 [19:37:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 3.95, 3.92 [19:38:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.44, 9.04, 9.66 [19:39:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.52, 10.13, 10.23 [19:40:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.55, 10.71, 10.20 [19:41:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.29, 11.38, 10.69 [19:42:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 11.01, 10.40 [19:43:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 4.00, 3.98 [19:43:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.74, 10.89, 10.58 [19:45:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 4.54, 4.20 [19:46:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.62, 10.81 [19:47:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.80, 5.92, 4.28 [19:47:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.67, 11.33, 10.76 [19:48:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.97, 3.38, 3.88 [19:48:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 11.48, 10.86 [19:49:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.52, 11.19, 10.77 [19:50:51] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.48, 3.35, 2.93 [19:52:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.73, 3.87 [19:52:50] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.98, 3.10, 2.88 [19:53:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.60, 9.06, 10.07 [19:54:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.03, 3.59 [19:55:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.61, 3.93 [19:57:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.77, 3.94 [19:57:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.44, 10.47, 10.43 [19:58:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.50, 3.63 [20:00:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.44, 3.59 [20:01:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.50, 6.54, 6.59 [20:01:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.77, 3.89 [20:01:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.14, 11.98, 11.01 [20:02:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.75, 10.95, 10.58 [20:04:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.21, 3.85, 3.69 [20:06:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.64, 3.63 [20:06:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.07, 11.52, 10.92 [20:08:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.61, 3.60 [20:09:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.62, 3.75 [20:09:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 11.91, 11.48 [20:10:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.84, 12.28, 11.28 [20:11:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.69, 3.75 [20:12:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.47, 3.57 [20:13:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.88, 3.80 [20:14:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 3.84, 3.68 [20:15:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.23, 6.77, 6.15 [20:15:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.85, 3.80 [20:15:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.05, 11.00, 11.11 [20:16:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 12.00, 11.89, 11.45 [20:18:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.33, 12.05, 11.56 [20:19:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.44, 11.32, 11.24 [20:20:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.66, 3.63 [20:22:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.52, 3.56 [20:22:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.68, 11.46, 11.48 [20:24:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.50, 3.55 [20:25:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.42, 4.06, 3.81 [20:26:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 4.32, 3.86 [20:27:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.36, 7.00, 7.20 [20:27:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.51, 11.47, 11.27 [20:28:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.76, 1.60 [20:29:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.64, 3.71 [20:29:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.54, 11.12, 11.15 [20:30:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.92, 11.55, 11.44 [20:30:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.46, 1.51 [20:31:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.15, 5.76, 6.63 [20:31:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.65, 3.70 [20:31:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.33, 11.73, 11.37 [20:32:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.84, 3.82 [20:32:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.24, 10.97, 11.24 [20:34:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.89, 1.75 [20:35:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.24, 3.51 [20:36:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.52, 1.63 [20:37:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.83, 6.65, 6.63 [20:37:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.65, 3.02, 3.39 [20:37:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.55, 11.26, 11.41 [20:39:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.18, 7.36, 6.88 [20:39:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.85, 11.96, 11.65 [20:40:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.42, 11.32 [20:41:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.76, 7.48, 6.99 [20:41:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.62, 11.50, 11.50 [20:43:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.18, 7.73, 7.14 [20:43:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.80, 3.62 [20:44:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.70, 3.68 [20:44:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.70, 11.47, 11.36 [20:45:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.27, 6.89, 6.88 [20:46:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.54, 3.64 [20:46:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.12, 11.79, 11.50 [20:47:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.42, 6.43, 6.69 [20:47:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.71, 3.68 [20:48:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.64, 11.94, 11.60 [20:51:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 11.11, 9.01, 7.69 [20:52:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.95, 12.13, 11.76 [20:53:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.49, 12.07, 11.72 [20:54:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.56, 4.39, 3.85 [20:54:50] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.09, 3.40 [20:55:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.27, 11.82, 11.68 [20:56:22] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:57:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.73, 12.34, 11.87 [20:58:22] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.811 second response time [20:58:59] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.633 second response time [20:59:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.43, 7.15, 7.54 [21:01:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.81, 10.46, 9.75 [21:03:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.75, 5.30, 6.69 [21:03:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.59, 3.43 [21:04:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.08, 10.12 [21:05:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.74, 9.83, 9.06 [21:05:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 3.10, 3.26 [21:06:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 10.86, 10.18 [21:07:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.83, 9.64, 9.10 [21:09:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.88, 2.92, 2.05 [21:10:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.15, 10.11, 10.05 [21:11:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 10.19, 9.45 [21:12:04] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:13:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.01, 9.85, 9.41 [21:16:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.69, 2.00, 1.97 [21:17:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 4.14, 3.67 [21:21:40] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.87, 1.90 [21:23:45] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.963 second response time [21:25:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.91, 3.77 [21:26:10] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 1.64, 1.82 [21:27:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 11.27, 10.38 [21:27:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.39, 4.31, 3.92 [21:28:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.98, 1.93 [21:29:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 10.54, 10.23 [21:29:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.79, 3.77 [21:29:37] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:30:18] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 9.52, 9.89 [21:31:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.49, 7.10, 6.15 [21:31:36] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.992 second response time [21:32:21] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.854 second response time [21:33:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.96, 7.51, 6.43 [21:33:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.83, 3.79 [21:35:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.58, 3.70 [21:37:22] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.49, 7.55, 6.66 [21:37:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.82, 4.31, 3.94 [21:38:53] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:39:22] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.68, 7.10, 6.60 [21:39:24] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:39:28] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:39:31] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.76, 1.97 [21:39:59] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:04] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [21:40:58] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:41:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.07, 6.48, 6.43 [21:41:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.66, 11.01, 11.92 [21:42:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 10.86, 10.24 [21:43:02] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 9.319 second response time [21:43:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.04, 9.79, 8.64 [21:43:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.74, 3.90 [21:43:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.70, 11.58, 12.01 [21:44:09] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.547 second response time [21:44:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.66, 3.94 [21:44:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 10.49, 10.19 [21:45:18] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.699 second response time [21:45:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.94, 1.93 [21:45:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.67, 9.26, 8.58 [21:45:33] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.137 second response time [21:45:41] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.553 second response time [21:46:12] [02miraheze/WikiDiscover] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/commit/0c4452fafb71 [21:46:13] [02miraheze/WikiDiscover] 07dependabot[bot] 030c4452f - Bump grunt-banana-checker from 0.9.0 to 0.10.0 [21:46:15] [02WikiDiscover] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 - 13https://github.com/miraheze/WikiDiscover [21:46:15] [url] Page not found · GitHub · GitHub | github.com [21:46:16] [02WikiDiscover] 07dependabot[bot] opened pull request 03#84: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/WikiDiscover/pull/84 [21:46:17] ... [21:46:18] [02WikiDiscover] 07dependabot[bot] labeled pull request 03#84: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/WikiDiscover/pull/84 [21:46:18] [url] Page not found · GitHub · GitHub | github.com [21:46:19] [02WikiDiscover] 07dependabot[bot] labeled pull request 03#84: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/WikiDiscover/pull/84 [21:46:19] ... [21:46:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.80, 10.04, 10.06 [21:47:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.45, 11.47, 11.90 [21:49:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.51, 1.97, 1.96 [21:49:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.21, 12.51, 12.22 [21:49:54] miraheze/WikiDiscover - dependabot[bot] the build passed. [21:51:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.51, 2.42, 2.12 [21:51:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.65, 2.77, 3.32 [21:52:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.27, 3.60 [21:54:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.24, 3.54 [21:55:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.01, 10.32, 10.22 [21:55:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.44, 3.46 [21:56:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.17, 1.81, 1.20 [21:57:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 10.12, 10.17 [21:57:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.58, 3.51 [21:58:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.46, 1.14 [21:59:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.95, 3.67 [22:00:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.53, 3.54 [22:01:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 10.40, 9.51 [22:01:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 4.23, 3.80 [22:02:41] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:43] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:29] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:22] [02WikiDiscover] 07Universal-Omega closed pull request 03#84: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/WikiDiscover/pull/84 [22:04:22] [url] Page not found · GitHub · GitHub | github.com [22:04:23] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/0bd18ba50e89...76c8b43d52ad [22:04:24] [url] Comparing 0bd18ba50e89...76c8b43d52ad · miraheze/WikiDiscover · GitHub | github.com [22:04:25] [02miraheze/WikiDiscover] 07dependabot[bot] 0376c8b43 - Bump grunt-banana-checker from 0.9.0 to 0.10.0 (#84) [22:04:25] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:29] [02miraheze/WikiDiscover] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 [22:04:30] [02WikiDiscover] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 - 13https://github.com/miraheze/WikiDiscover [22:04:31] [url] Page not found · GitHub · GitHub | github.com [22:04:42] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.742 second response time [22:04:44] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.832 second response time [22:05:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.56, 1.91 [22:05:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.79, 3.75 [22:05:36] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.515 second response time [22:06:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.92, 3.89 [22:06:32] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.206 second response time [22:06:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.38, 10.41, 10.26 [22:07:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.81, 3.75 [22:07:58] miraheze/WikiDiscover - Universal-Omega the build passed. [22:08:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.05, 3.93 [22:08:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.40, 9.98, 10.12 [22:09:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.47, 10.13, 9.86 [22:09:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.55, 1.74, 1.87 [22:09:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.64, 3.69 [22:10:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.73, 3.83 [22:11:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 2.00, 1.97 [22:13:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 10.28, 10.01 [22:14:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 10.88, 10.43 [22:15:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 2.91, 3.33 [22:16:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 10.51, 10.36 [22:17:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.91, 1.30, 1.66 [22:20:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 2.91, 3.35 [22:20:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.25, 10.13, 10.09 [22:23:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.97, 1.84 [22:23:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.35, 3.32 [22:24:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.44, 3.51 [22:25:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.55, 1.70 [22:25:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.35, 3.31 [22:26:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.79, 3.63 [22:27:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.51, 2.33, 1.75 [22:28:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.73, 3.64 [22:28:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.09, 10.55 [22:29:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.08, 1.91, 1.66 [22:29:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.55, 3.42 [22:30:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 10.94, 10.55 [22:31:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.16, 3.29 [22:34:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 4.08, 3.82 [22:34:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 11.50, 10.86 [22:35:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.47, 1.55 [22:36:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.57, 3.53, 3.65 [22:36:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 11.62, 10.98 [22:39:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.64, 2.00 [22:39:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.61, 10.61, 10.23 [22:40:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.09, 2.94, 3.38 [22:40:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.62, 11.07 [22:41:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.45, 9.90, 10.01 [22:41:47] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:14] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:15] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:24] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.90, 10.37, 10.69 [22:42:48] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:48] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.809 second response time [22:44:13] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.782 second response time [22:44:14] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.792 second response time [22:44:21] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/fe8d0cc995c2 [22:44:22] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.621 second response time [22:44:23] [02miraheze/CreateWiki] 07dependabot[bot] 03fe8d0cc - Bump grunt-banana-checker from 0.9.0 to 0.10.0 [22:44:24] [02CreateWiki] 07dependabot[bot] opened pull request 03#327: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/CreateWiki/pull/327 [22:44:24] [url] Page not found · GitHub · GitHub | github.com [22:44:26] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 - 13https://github.com/miraheze/CreateWiki [22:44:26] ... [22:44:27] [02CreateWiki] 07dependabot[bot] labeled pull request 03#327: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/CreateWiki/pull/327 [22:44:27] [url] Page not found · GitHub · GitHub | github.com [22:44:29] [02CreateWiki] 07dependabot[bot] labeled pull request 03#327: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/CreateWiki/pull/327 [22:44:29] ... [22:44:55] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.621 second response time [22:46:42] [02miraheze/IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/commit/ffaa4191f95d [22:46:44] [02miraheze/IncidentReporting] 07dependabot[bot] 03ffaa419 - Bump grunt-banana-checker from 0.9.0 to 0.10.0 [22:46:45] [02IncidentReporting] 07dependabot[bot] opened pull request 03#45: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/IncidentReporting/pull/45 [22:46:46] [url] Page not found · GitHub · GitHub | github.com [22:46:47] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/grunt-banana-checker-0.10.0 - 13https://github.com/miraheze/IncidentReporting [22:46:47] ... [22:46:48] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#45: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/IncidentReporting/pull/45 [22:46:48] [url] Page not found · GitHub · GitHub | github.com [22:46:50] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#45: Bump grunt-banana-checker from 0.9.0 to 0.10.0 - 13https://github.com/miraheze/IncidentReporting/pull/45 [22:49:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.58, 2.05, 1.98 [22:50:02] miraheze/IncidentReporting - dependabot[bot] the build passed. [22:50:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.58, 11.10, 10.67 [22:52:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.35, 3.25 [22:54:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 11.04, 10.79 [22:54:26] miraheze/CreateWiki - dependabot[bot] the build passed. [22:56:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [22:57:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.95, 1.99 [22:57:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 3.67, 3.18 [22:57:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 2.15, 1.66 [22:58:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.39, 4.03, 3.51 [22:59:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.73, 3.27 [22:59:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.61, 1.52 [23:00:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 3.32, 3.31 [23:04:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.97, 3.59 [23:05:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.68, 3.36 [23:06:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.84, 3.58 [23:07:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.55, 3.36 [23:08:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.89, 3.63 [23:10:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.88, 3.66 [23:12:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.58, 8.77, 9.83 [23:15:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.89, 1.89 [23:17:06] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.96, 1.91 [23:18:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.00, 4.02, 3.77 [23:18:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.75, 3.53 [23:19:06] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.42, 2.51, 2.12 [23:22:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.35, 3.45 [23:24:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.42, 3.44 [23:25:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.12, 9.96, 9.40 [23:26:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.36, 3.41 [23:28:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 10.73, 10.10 [23:28:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.75, 3.55 [23:29:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.60, 9.75, 9.47 [23:29:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.71, 2.16, 1.76 [23:30:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.80, 3.60 [23:32:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.35, 9.55, 9.78 [23:32:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.44, 3.03, 3.35 [23:34:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.50, 3.61, 3.91 [23:35:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 11.04, 10.04 [23:36:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.59, 10.12 [23:38:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 4.15, 4.02 [23:38:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.72, 9.77, 9.87 [23:39:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 9.78, 9.78 [23:40:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.84, 4.03, 3.63 [23:41:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.50, 9.93, 8.98 [23:42:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.96, 3.98 [23:43:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.74, 1.98, 1.99 [23:44:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 4.37, 4.13 [23:45:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.73, 9.50, 9.06 [23:45:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 11.26, 11.98 [23:46:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [23:48:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.67, 3.66 [23:49:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.33, 2.06, 1.68 [23:49:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.37, 11.86, 12.04 [23:50:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 4.00, 3.77 [23:51:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.09, 11.47, 11.87 [23:54:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 11.03, 11.96 [23:55:49] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:49] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.758 second response time [23:59:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.76, 1.77 [23:59:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds