[00:01:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.21, 9.10, 8.29 [00:04:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.42, 8.58 [00:04:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 9.48, 8.60 [00:04:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.77, 10.18, 8.69 [00:05:26] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:07:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.90, 9.31, 8.69 [00:07:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.62, 10.78, 9.29 [00:10:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.82, 10.11, 9.11 [00:11:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.30, 9.63, 8.69 [00:12:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.59, 9.88, 8.43 [00:12:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.84, 9.44, 8.32 [00:14:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 9.79, 8.94 [00:15:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.34, 9.90, 8.73 [00:21:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.60, 9.70, 8.93 [00:21:46] [02mw-config] 07Universal-Omega edited pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:23:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.58, 9.19 [00:24:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.37, 9.24, 8.87 [00:24:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.19, 9.58, 8.86 [00:25:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.57, 9.48, 9.73 [00:26:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.47, 3.14 [00:26:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.79, 9.91, 9.40 [00:27:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 8.98, 8.74 [00:28:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.13, 9.87, 9.98 [00:28:49] [02mw-config] 07Universal-Omega edited pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:29:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 2.97, 2.98 [00:31:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.60, 9.65 [00:32:30] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.05, 9.59, 9.95 [00:34:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 9.27, 9.54 [00:36:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.41, 3.28 [00:37:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.95, 9.88 [00:39:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.07, 3.18 [00:39:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.40, 9.03 [00:39:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.70, 11.00, 10.39 [00:40:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.89, 10.61, 10.14 [00:40:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 9.52, 9.60 [00:41:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.53, 10.59, 9.97 [00:45:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.79, 3.48 [00:46:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 9.80, 9.91 [00:46:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 10.29, 9.91 [00:48:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.33, 8.92, 9.21 [00:49:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.30, 3.36 [00:49:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 10.00, 9.86 [00:50:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 9.79, 9.93 [00:51:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.44, 9.94, 9.40 [00:52:21] [02mw-config] 07Universal-Omega edited pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [00:52:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.24, 10.05, 9.87 [00:54:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.68, 10.04, 9.60 [00:54:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.25, 8.66, 9.00 [00:55:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 9.55, 9.72 [00:55:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 9.68, 9.75 [00:57:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.90, 9.08, 9.31 [00:58:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.45, 9.80, 9.80 [00:58:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.06, 10.91, 10.20 [01:00:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.11, 9.15, 9.26 [01:01:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.21, 9.98, 9.85 [01:03:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.37, 9.37, 9.34 [01:04:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.60, 9.79, 9.79 [01:05:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.51, 2.01, 1.53 [01:06:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 10.00, 9.61 [01:06:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.37, 7.85, 8.40 [01:07:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.81, 10.40, 10.03 [01:08:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.41, 1.38 [01:11:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.40, 9.87, 9.49 [01:14:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.82, 8.68, 8.69 [01:14:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 9.86, 9.55 [01:16:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 9.37, 9.73 [01:17:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.62, 7.62, 8.27 [01:18:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 9.52, 9.75 [01:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:28:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.55, 9.24, 9.38 [01:29:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.15, 9.14, 9.89 [01:31:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.01, 8.57, 9.14 [01:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:34:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.08, 9.30, 9.61 [01:35:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 8.69, 8.78 [01:36:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.79, 9.05, 9.83 [01:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.93, 9.23, 9.51 [01:39:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.63, 8.18, 8.57 [01:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:40:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 10.11, 9.82 [01:41:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.64, 3.29, 2.60 [01:41:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.59, 7.74, 8.40 [01:43:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.38, 8.82, 9.37 [01:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:46:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 9.21, 9.21 [01:46:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.46, 3.01 [01:48:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 8.98, 9.15 [01:49:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 8.86, 9.05 [01:49:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.74, 3.20 [01:51:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.74, 7.75, 8.30 [01:51:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 8.94, 9.16 [01:55:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.81, 3.39 [01:55:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 7.80, 8.48 [01:55:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.67, 9.55, 9.34 [01:56:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.53, 1.93 [01:58:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.58, 3.36 [01:58:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.03, 9.02, 9.14 [02:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:01:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.56, 4.58, 3.76 [02:01:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.42, 10.03, 9.50 [02:02:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.83, 9.08, 8.89 [02:04:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.84, 3.20 [02:04:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.38, 8.92, 8.25 [02:06:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.15, 9.70, 9.28 [02:07:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.35, 3.13 [02:07:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.78, 7.36, 7.77 [02:08:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.55, 9.55, 9.13 [02:10:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.47, 3.62 [02:11:56] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.45, 6.62, 4.62 [02:11:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 9.32, 9.12 [02:12:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.32, 1.64 [02:15:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.43, 9.78, 9.29 [02:17:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.82, 2.96, 3.38 [02:18:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.75, 8.23, 8.81 [02:21:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.31, 7.25, 8.31 [02:25:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.02, 8.28, 9.31 [02:25:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.60, 1.91, 1.68 [02:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:27:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.16, 4.44, 3.83 [02:28:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.25, 9.31, 9.48 [02:28:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.57, 8.21, 8.32 [02:31:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.88, 9.83, 9.68 [02:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.58, 1.32, 1.51 [02:32:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.11, 8.74, 9.79 [02:33:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.52, 3.65 [02:35:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.19, 10.33, 10.16 [02:37:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.53, 9.54 [02:39:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.76, 3.36 [02:40:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.34, 8.12, 8.25 [02:42:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.89, 3.46 [02:45:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.66, 3.44 [02:45:56] PROBLEM - mon111 Current Load on mon111 is WARNING: WARNING - load average: 5.63, 4.92, 4.33 [02:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:48:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.20, 3.29 [02:48:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 2.98, 3.32 [02:49:05] RECOVERY - mon111 Current Load on mon111 is OK: OK - load average: 4.97, 5.08, 4.51 [02:56:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:59:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.95, 3.27, 3.43 [03:02:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.03, 3.31 [03:08:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.34, 8.51, 9.58 [03:22:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.84, 4.84, 3.87 [03:25:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 4.02, 3.22 [03:25:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 4.15, 3.58 [03:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:28:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.92, 3.33 [03:28:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.77, 3.53 [03:31:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.30, 3.60 [03:32:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.87, 3.61 [03:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:38:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.05, 3.40, 3.49 [03:38:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.43, 3.71, 3.72 [03:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:39:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.55, 9.96, 9.26 [03:41:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.64, 3.16 [03:41:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.36, 3.87 [03:44:35] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.72, 2.16, 1.67 [03:45:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 2.82, 3.31 [03:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.85, 1.77, 1.63 [03:50:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.95, 2.19, 1.80 [03:54:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.36, 1.87, 1.75 [03:54:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 2.79, 3.27 [03:57:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.58, 3.74, 2.51 [04:03:00] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.97, 2.74, 1.85 [04:03:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.62, 1.73, 2.00 [04:08:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 9.08, 9.70 [04:10:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.01, 1.56 [04:11:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.61, 10.72, 10.25 [04:14:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.43, 8.27, 7.26 [04:17:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.07, 7.59, 7.22 [04:30:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.58, 9.30, 9.93 [04:42:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.62, 9.39, 9.42 [05:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:04:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.53, 8.73, 9.75 [05:10:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.22, 9.52 [05:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:13:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 9.51, 9.62 [05:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:17:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.18, 1.17, 1.92 [05:19:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.21, 9.42, 9.58 [05:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.49, 9.59 [05:23:04] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.47, 1.01, 1.63 [05:25:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.58, 10.20, 9.87 [05:28:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.59, 8.33, 7.26 [05:31:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 8.97, 7.69 [05:34:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.29, 1.71, 1.55 [05:34:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 9.40, 8.10 [05:37:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.55, 1.53 [05:38:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 8.59, 7.50 [05:40:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.08, 8.14, 7.95 [05:41:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.07, 9.24, 7.93 [05:44:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.80, 8.19, 7.75 [05:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:46:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.65, 9.12, 9.77 [05:49:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.94, 9.89, 9.91 [05:51:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:52:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.01, 9.21, 9.66 [05:55:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.13, 8.28, 9.85 [05:56:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:58:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.50, 8.68, 9.13 [06:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:01:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 8.13, 8.84 [06:01:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.66, 10.00 [06:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.39, 9.21 [06:12:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.67, 8.15, 8.76 [06:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.07, 9.09, 8.89 [06:28:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 9.76, 9.63 [06:31:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.09, 10.42, 9.89 [06:40:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 9.23, 9.46 [06:55:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.91, 9.61, 9.31 [06:58:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 9.61, 9.36 [07:02:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.02, 7.31, 9.48 [07:03:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.67, 7.37, 8.42 [07:05:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.12, 8.27, 9.40 [07:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:08:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.02, 8.07, 9.13 [07:08:46] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.51, 4.03, 7.83 [07:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:14:14] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.48, 2.35, 5.94 [07:14:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:16:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.12, 7.11, 8.25 [08:15:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:02:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.37, 2.50 [09:04:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.98, 2.95 [09:07:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 3.28, 2.86 [09:08:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.01, 2.62 [09:18:47] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.04, 2.15, 1.13 [09:32:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.40, 1.87, 1.71 [09:35:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.15, 1.54, 1.61 [10:24:52] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.83, 1.37, 1.01 [10:27:38] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.94, 1.27, 1.04 [10:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:49:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.36, 1.76, 1.40 [11:52:20] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.72, 2.45, 1.71 [12:00:59] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.61, 1.69 [12:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:22:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 8.95, 8.09 [12:25:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.41, 10.19, 8.74 [12:27:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 8.25, 7.47 [12:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:28:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.89, 9.89, 8.85 [12:36:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.40, 8.35, 7.89 [12:39:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.19, 9.10, 8.59 [12:43:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 8.06, 8.50 [12:45:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.13, 8.19, 8.40 [12:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:50:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.28, 9.59, 9.17 [12:55:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.98, 7.14, 8.18 [13:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:29:22] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/101f757d91ff...171858752535 [13:29:23] [02miraheze/puppet] 07JohnFLewis 031718587 - varnish: improve X-Cache header [13:36:34] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.98, 2.49, 1.74 [13:41:27] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/171858752535...6b040a6605b3 [13:41:29] [02miraheze/puppet] 07JohnFLewis 036b040a6 - varnish: set x-Cache in request as well as response [13:42:28] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.61, 1.62 [13:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.38, 9.07, 8.02 [13:50:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.43, 9.74, 8.49 [13:53:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.38, 8.93 [13:56:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.49, 9.56, 8.86 [14:01:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.14, 8.34 [14:04:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:05:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.32, 7.22, 8.09 [14:07:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.53, 7.63, 7.95 [14:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:12:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 8.35, 8.26 [14:15:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.73, 7.50, 7.95 [14:17:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.64, 3.30 [14:20:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 3.99, 3.45 [14:22:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 8.32, 8.12 [14:23:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.76, 3.44 [14:25:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 8.92, 8.37 [14:27:43] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 12.15, 9.20, 5.77 [14:27:46] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.13, 1.96, 1.76 [14:28:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.08, 8.54, 8.36 [14:29:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.41, 3.37 [14:30:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.53, 1.62, 1.65 [14:31:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.29, 8.13, 8.22 [14:31:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 9.70, 9.13 [14:32:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.91, 3.18 [14:34:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.08, 9.75 [14:39:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.96, 8.48, 7.76 [14:41:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.14, 9.00, 8.15 [14:44:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.28, 8.64, 8.13 [14:46:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.24, 7.49, 7.65 [14:47:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.82, 7.91, 7.99 [14:47:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.09, 10.59, 9.41 [14:48:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.07, 9.73, 8.81 [14:51:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.16, 8.07, 8.33 [15:10:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.79, 9.01, 9.98 [15:11:19] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.237 second response time [15:12:23] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.683 second response time [15:18:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.99, 9.73, 9.95 [15:18:30] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.041 second response time [15:19:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.69, 1.81, 1.54 [15:22:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.39, 1.57, 1.49 [15:30:34] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.73, 1.66 [15:33:34] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.24, 1.58, 1.62 [15:36:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 9.36, 9.90 [15:43:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.91, 10.35, 10.15 [15:58:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.00, 8.90, 9.97 [16:02:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.87, 10.10, 10.25 [16:05:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.74, 9.34, 9.94 [16:14:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.49, 8.91, 9.36 [16:18:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 8.31, 7.90 [16:21:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.58, 7.67, 7.73 [16:21:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:32:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.09, 1.59, 1.43 [16:35:33] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.09, 1.41, 1.39 [16:36:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 9.52, 9.92 [16:44:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.47, 3.13 [16:47:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 4.17, 3.44 [16:48:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.04, 9.65, 9.65 [16:50:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 3.29, 3.23 [16:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.76, 9.44, 9.77 [16:58:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 9.30, 9.93 [16:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:00:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.76, 1.48 [17:01:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.63, 9.96 [17:04:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.47, 1.42 [17:04:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:09:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:17:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:18:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.24, 2.96 [17:18:52] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:13] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 9.72, 9.38 [17:21:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 3.17, 2.99 [17:22:05] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.011 second response time [17:22:09] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.291 second response time [17:22:27] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.063 second response time [17:32:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 9.64, 9.89 [17:35:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 10.05, 10.00 [17:36:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.48, 8.67, 7.95 [17:39:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.02, 8.35, 7.95 [17:44:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.08, 8.74, 8.16 [17:50:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.51, 7.47, 7.83 [17:55:05] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [17:55:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/028aee7fd8b5...d203a6436ba4 [17:55:08] [02miraheze/mw-config] 07Universal-Omega 03d203a64 - Set sitename and server for all wikis [17:56:14] miraheze/mw-config - Universal-Omega the build has errored. [17:56:59] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:02:08] Are the MW team aware there’s consistently thousands of jobs in the jobqueue and the number looks to be hanging around 2-3k jobs? [18:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:03:07] JohnLewis: no [18:03:21] Has rdns been added to mail [18:03:39] I had an email about email to Doug being rejected [18:03:50] paladox ^ [18:04:32] i don't think rdns works at least when i tried to add it to dns it didn't work JohnLewis RhinosF1 [18:04:53] paladox: what do you mean doesn't work [18:05:09] exactly what i said. [18:05:21] No hostname is returned [18:06:34] Right [18:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:06:47] CosmicAlpha: I think https://github.com/miraheze/puppet/commit/04c723868d23d89ee882d507801a9a83dd79ab0e#diff-40c068c3583e3cd1e27bee85a74e4a5f8683542b76688c9598b41b70c4b23084 needs reverting [18:06:48] [url] jobrunner: add low-priority jobs (#2382) · miraheze/puppet@04c7238 · GitHub | github.com [18:07:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 4.59, 3.65 [18:08:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:59] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:38] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:05] Fun [18:10:50] It's locked up [18:10:51] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.74, 5.30, 4.30 [18:11:18] JohnLewis: can you restart php pls on ^ [18:11:30] paladox: rDNS is a requirement for mail [18:11:41] It needs fixing as mail is being rejected [18:12:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 8.410 second response time [18:13:28] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.695 second response time [18:13:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.45, 8.87, 8.04 [18:13:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.45, 3.50 [18:14:31] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.030 second response time [18:15:10] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.307 second response time [18:16:00] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.828 second response time [18:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:17:07] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-1/±0] 13https://github.com/miraheze/dns/commit/df16eb2917cb [18:17:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.41, 3.27 [18:17:09] [02miraheze/dns] 07paladox 03df16eb2 - Fix typo [18:17:10] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [18:17:12] [02dns] 07paladox opened pull request 03#251: Fix typo - 13https://github.com/miraheze/dns/pull/251 [18:17:42] [02dns] 07paladox closed pull request 03#251: Fix typo - 13https://github.com/miraheze/dns/pull/251 [18:17:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/dns/compare/230781429d30...83c34d28f9b0 [18:17:45] [02miraheze/dns] 07paladox 0383c34d2 - Fix typo (#251) [18:17:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.94, 3.48, 3.85 [18:17:54] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [18:17:55] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [18:18:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/42b95308970f [18:18:21] [02miraheze/puppet] 07paladox 0342b9530 - base::dns: resolv search 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [18:18:23] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [18:18:24] [02puppet] 07paladox opened pull request 03#2404: base::dns: resolv search 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa - 13https://github.com/miraheze/puppet/pull/2404 [18:18:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.01, 3.11, 3.66 [18:19:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.17, 7.41, 7.69 [18:20:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 2.93, 3.10 [18:20:42] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:21:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.32, 3.68 [18:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:24:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.75, 3.38 [18:24:37] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [18:25:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 2.87, 3.37 [18:26:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ee18ad4c41aa [18:26:22] [02miraheze/puppet] 07paladox 03ee18ad4 - base::dns: Look for 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa in forward-zones [18:26:24] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [18:26:25] [02puppet] 07paladox opened pull request 03#2405: base::dns: Look for 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa in forward-zones - 13https://github.com/miraheze/puppet/pull/2405 [18:26:50] [02puppet] 07paladox closed pull request 03#2404: base::dns: resolv search 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa - 13https://github.com/miraheze/puppet/pull/2404 [18:26:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [18:26:53] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [18:27:31] [02puppet] 07paladox closed pull request 03#2405: base::dns: Look for 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa in forward-zones - 13https://github.com/miraheze/puppet/pull/2405 [18:27:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [18:27:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b040a6605b3...b4fc7e9f84a9 [18:27:36] [02miraheze/puppet] 07paladox 03b4fc7e9 - base::dns: Look for 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa in forward-zones (#2405) [18:27:37] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [18:30:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 4.26, 2.92 [18:32:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.18, 3.41 [18:35:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.14, 3.14, 3.34 [18:37:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.26, 8.62, 9.90 [18:38:24] ok so, I still can't access my wiki [18:38:52] trying to avoid creating a phab task because I believe the error is just an oversight on the server side [18:39:11] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/b711eba8b0eb [18:39:12] [02miraheze/dns] 07paladox 03b711eba - Add puppet111 ptr [18:39:14] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [18:39:15] [02dns] 07paladox opened pull request 03#252: Add puppet111 ptr - 13https://github.com/miraheze/dns/pull/252 [18:39:20] [02dns] 07paladox closed pull request 03#252: Add puppet111 ptr - 13https://github.com/miraheze/dns/pull/252 [18:39:22] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/83c34d28f9b0...2ee90d97d1a6 [18:39:23] [02miraheze/dns] 07paladox 032ee90d9 - Add puppet111 ptr (#252) [18:39:25] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [18:39:26] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [18:41:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.61, 10.05, 10.21 [18:43:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.18, 1.24, 1.83 [18:46:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.16, 1.68 [18:49:40] [02puppet] 07Universal-Omega opened pull request 03#2406: Revert "jobrunner: add low-priority jobs" - 13https://github.com/miraheze/puppet/pull/2406 [18:50:51] RhinosF1: ^ [18:52:39] I just noticed now [18:52:49] the error is "exception caught during exception handler" [18:52:50] LOL [18:52:54] (oops,caps) [18:55:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d203a6436ba4...c4c79c253f2f [18:55:09] [02miraheze/mw-config] 07Universal-Omega 03c4c79c2 - Fix syntax + fix support for deleted databases [18:55:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [18:55:30] CosmicAlpha could you check this for me? [18:55:37] (when you have time, of course) [18:55:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.56, 3.28 [18:56:16] miraheze/mw-config - Universal-Omega the build passed. [18:56:54] Lakelimbo: what's the exception ID (the hash of characters)? I'm currently mobile but I can look in a little bit. [18:57:24] ca27c5440351229dc5a8eaee [18:57:31] Fatal exception of type "Error" [18:58:11] Thanks! [18:58:18] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/dns/commit/4e2a8c8c9734 [18:58:19] [02miraheze/dns] 07paladox 034e2a8c8 - Add mail121 ptr [18:58:21] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [18:58:22] [02dns] 07paladox opened pull request 03#253: Add mail121 ptr - 13https://github.com/miraheze/dns/pull/253 [18:59:03] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4e2a8c8c9734...ebebf2c47df5 [18:59:04] [02miraheze/dns] 07paladox 03ebebf2c - Update 139.105.24.31.in-addr.arpa [18:59:06] [02dns] 07paladox synchronize pull request 03#253: Add mail121 ptr - 13https://github.com/miraheze/dns/pull/253 [18:59:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.27, 3.20 [19:00:00] [02dns] 07paladox closed pull request 03#253: Add mail121 ptr - 13https://github.com/miraheze/dns/pull/253 [19:00:01] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/2ee90d97d1a6...0ab2cea076d4 [19:00:03] [02miraheze/dns] 07paladox 030ab2cea - Add mail121 ptr (#253) [19:00:04] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [19:00:06] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:00:43] paladox: can you merge https://github.com/miraheze/puppet/pull/2406 [19:00:44] [url] Revert "jobrunner: add low-priority jobs" by Universal-Omega · Pull Request #2406 · miraheze/puppet · GitHub | github.com [19:00:53] CosmicAlpha: we might need to manually run jobs [19:01:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b4fc7e9f84a9...5e35ce4defcb [19:01:05] [02miraheze/puppet] 07paladox 035e35ce4 - base::dns: look for 139.105.24.31.in-addr.arpa in forward-zones [19:01:15] [02puppet] 07paladox closed pull request 03#2406: Revert "jobrunner: add low-priority jobs" - 13https://github.com/miraheze/puppet/pull/2406 [19:01:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/5e35ce4defcb...19f401306620 [19:01:18] [02miraheze/puppet] 07Universal-Omega 0319f4013 - Revert "jobrunner: add low-priority jobs (#2382)" (#2406) [19:02:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c4c79c253f2f...cfd198997a68 [19:02:05] [02miraheze/mw-config] 07Universal-Omega 03cfd1989 - Add $cacheArray to properties [19:02:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [19:03:12] miraheze/mw-config - Universal-Omega the build has errored. [19:05:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cfd198997a68...3ea3071ddb9b [19:05:24] [02miraheze/mw-config] 07Universal-Omega 033ea3071 - Update .phpcs.xml [19:05:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [19:06:17] [02mw-config] 07Universal-Omega edited pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [19:06:35] miraheze/mw-config - Universal-Omega the build passed. [19:07:35] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/b07015b50400 [19:07:37] [02miraheze/dns] 07paladox 03b07015b - Add mail121 ipv6 ptr [19:07:38] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:07:40] [02dns] 07paladox opened pull request 03#254: Add mail121 ipv6 ptr - 13https://github.com/miraheze/dns/pull/254 [19:07:47] [02dns] 07paladox closed pull request 03#254: Add mail121 ipv6 ptr - 13https://github.com/miraheze/dns/pull/254 [19:07:48] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0ab2cea076d4...89cabb9dea8b [19:07:50] [02miraheze/dns] 07paladox 0389cabb9 - Add mail121 ipv6 ptr (#254) [19:07:51] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [19:07:53] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [19:08:08] RhinosF1: added mail121 ptr [19:11:20] Ty [19:17:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:23:25] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.77, 3.17, 7.22 [19:23:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.84, 7.63, 9.48 [19:24:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.62, 8.81, 9.77 [19:26:13] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.52, 2.77, 6.39 [19:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:30:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.19, 2.98 [19:31:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 8.54, 7.84 [19:33:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.67, 2.81 [19:34:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.67, 8.93, 8.28 [19:34:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.86, 9.72, 8.40 [19:35:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.11, 9.88, 9.54 [19:37:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.41, 8.25, 8.14 [19:37:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.37, 8.51, 8.17 [19:38:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.39, 8.95, 9.19 [19:40:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.27, 7.56, 7.87 [19:43:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.45, 8.66, 7.78 [19:44:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.68, 9.39, 8.62 [19:46:51] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.29, 7.99, 7.67 [19:47:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.64, 8.47, 8.42 [19:47:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:47:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.65, 7.61, 8.41 [19:49:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.10, 9.80, 9.40 [19:52:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.14, 9.01, 9.20 [19:54:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 2.30, 1.66 [19:55:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.69, 2.27, 1.73 [19:57:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.72, 1.54, 1.47 [19:57:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.74, 7.54, 8.38 [19:58:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.85, 1.78, 1.64 [20:02:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.90, 2.48, 1.87 [20:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:05:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.67, 1.65 [20:07:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.85, 8.90, 8.43 [20:09:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.04, 4.15, 3.38 [20:10:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.13, 9.22, 8.61 [20:10:41] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/0eff922022f1 [20:10:42] [02miraheze/dns] 07paladox 030eff922 - Remove ns3 [20:10:44] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [20:10:45] [02dns] 07paladox opened pull request 03#255: Remove ns3 - 13https://github.com/miraheze/dns/pull/255 [20:12:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.34, 3.21 [20:12:06] [02dns] 07paladox closed pull request 03#255: Remove ns3 - 13https://github.com/miraheze/dns/pull/255 [20:12:07] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/89cabb9dea8b...a1d88f115583 [20:12:08] [02miraheze/dns] 07paladox 03a1d88f1 - Remove ns3 (#255) [20:12:10] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [20:12:11] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [20:13:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 8.56, 8.44 [20:15:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.58, 9.42, 8.12 [20:16:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 9.70, 8.94 [20:17:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 8.93, 8.48 [20:18:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.34, 10.53, 9.14 [20:18:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.78, 8.97, 8.01 [20:19:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 9.49, 9.00 [20:21:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 8.38, 7.97 [20:22:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.63, 10.26, 9.36 [20:23:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.95, 9.89, 8.98 [20:24:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.64, 9.53, 9.38 [20:24:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 9.09, 8.54 [20:27:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.68, 9.75, 9.47 [20:28:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 9.90, 9.53 [20:28:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.37, 8.78, 8.28 [20:29:11] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.09, 2.07, 1.76 [20:29:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.03, 9.16, 9.03 [20:30:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.18, 8.09, 8.26 [20:31:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.13, 9.84, 9.55 [20:31:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.31, 10.22, 8.87 [20:32:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.78, 10.35, 9.47 [20:35:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.57, 9.50, 8.85 [20:35:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.97, 1.61, 1.67 [20:38:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.46, 9.36, 8.79 [20:41:11] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 10.16, 9.39 [20:41:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.03, 10.85, 9.42 [20:43:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 9.06, 8.55 [20:44:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.88, 9.57, 9.26 [20:46:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.50, 9.35, 8.73 [20:50:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.44, 9.54, 8.94 [20:50:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.58, 9.56, 9.33 [20:50:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 9.50, 9.52 [20:53:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 9.51, 9.39 [20:55:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.48, 3.14 [20:57:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.81, 9.64, 9.53 [20:58:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.05, 3.29, 3.14 [20:59:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.06, 9.50, 9.38 [21:00:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.79, 9.20, 9.90 [21:02:44] [02puppet] 07paladox commented on pull request 03#2403: Addition of 29tb.github.io to CSP - 13https://github.com/miraheze/puppet/pull/2403#issuecomment-1025230419 [21:02:45] [02puppet] 07paladox closed pull request 03#2403: Addition of 29tb.github.io to CSP - 13https://github.com/miraheze/puppet/pull/2403 [21:03:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.47, 9.49, 9.41 [21:03:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.65, 9.88, 10.00 [21:05:17] Ty paladox [21:05:30] yw [21:05:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.88, 8.23, 8.50 [21:06:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.28, 8.14, 8.98 [21:06:41] thanks paladox :) [21:07:48] Thanks paladox [21:08:11] Have you disabled my Phab accounts? [21:09:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.25, 9.38, 9.31 [21:09:30] Also,could you archive my Bukkit project? [21:09:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 9.47, 9.29 [21:13:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.39, 9.39, 9.00 [21:15:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.02, 8.76, 9.10 [21:21:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.34, 8.96, 9.00 [21:23:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.95, 3.53 [21:24:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.67, 8.66, 8.91 [21:26:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.31, 7.83, 8.50 [21:30:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.50, 8.95, 8.89 [21:31:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.54, 8.74, 9.54 [21:31:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.29, 9.91 [21:32:04] @Bukkit: archived [21:32:14] Thanks [21:32:25] paladox: can you disable the accounts on https://phabricator.miraheze.org/project/members/73/ [21:32:26] [url] Login | phabricator.miraheze.org [21:32:59] And also 29TB [21:32:59] done [21:33:06] !log disable two accounts on phabricator [21:33:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:33:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.84, 8.72, 8.62 [21:33:22] That was fast [21:33:25] paladox: also https://phabricator.miraheze.org/p/29TB/ [21:33:26] [url] ♟ 29TB | phabricator.miraheze.org [21:33:39] done [21:33:46] !log disable one account on phabricator [21:33:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.82, 8.38, 8.69 [21:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.66, 10.16, 10.12 [21:35:07] Ty paladox [21:35:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 2.92, 3.25 [21:37:05] @Bukkit: I also did https://phab.mirahezebots.org/people/manage/25/#54 [21:37:06] [url] ♟ Manage User · Bukkit | phab.mirahezebots.org [21:38:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.73, 8.48, 9.64 [21:38:45] Oh, fortgot about that. Thanks :) [21:38:53] Forgot* [21:40:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.71, 8.98, 9.20 [21:40:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.70, 9.71, 9.90 [21:41:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 9.65, 9.84 [21:41:53] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.40, 2.40, 2.01 [21:41:55] Np [21:42:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.97, 7.91, 8.44 [21:43:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 8.90, 9.13 [21:44:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.00, 1.82, 1.86 [21:45:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.60, 8.00, 8.39 [21:46:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.35, 9.58, 9.31 [21:47:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.50, 3.03 [21:48:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.86, 3.51 [21:49:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.21, 8.57, 8.94 [21:50:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.44, 8.98, 9.56 [21:50:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.80, 2.84 [21:51:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.41, 1.69 [21:55:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 4.76, 4.03 [21:56:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.28, 7.61, 8.34 [21:56:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.87, 9.61, 9.42 [21:57:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:59:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.13, 9.28 [22:01:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 3.44, 3.74 [22:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:03:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.16, 8.14, 9.46 [22:04:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.77, 3.76, 3.79 [22:05:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.54, 9.66, 9.34 [22:07:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.01, 3.02, 3.50 [22:08:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.95, 3.54 [22:10:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 2.94, 3.40 [22:11:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.99, 9.32, 9.27 [22:14:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.09, 3.27 [22:15:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.80, 8.82, 8.52 [22:18:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.99, 7.49, 8.06 [22:18:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.50, 9.44, 9.30 [22:20:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.05, 9.05 [22:24:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.57, 7.65, 8.47 [22:25:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.39, 3.37 [22:27:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 2.26, 1.82 [22:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:27:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.48, 9.81, 9.74 [22:28:46] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 12.44, 9.45, 6.55 [22:30:31] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.53, 1.60 [22:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:34:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 3.27, 3.40 [22:40:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.27, 9.93, 9.52 [22:41:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.57, 8.04, 8.06 [22:43:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 9.87, 9.61 [22:45:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.72, 3.56 [22:47:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.69, 3.36 [22:47:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.50, 7.98, 8.05 [22:49:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.85, 10.21, 9.80 [22:51:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.81, 3.02, 3.36 [22:51:39] [02RemovePII] 07Universal-Omega closed pull request 03#41: 1.38 support - 13https://github.com/miraheze/RemovePII/pull/41 [22:51:42] [02RemovePII] 07Universal-Omega deleted branch 031_38-support - 13https://github.com/miraheze/RemovePII [22:51:43] [02miraheze/RemovePII] 07Universal-Omega deleted branch 031_38-support [22:52:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.72, 3.07, 2.32 [22:53:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.23, 3.30 [22:58:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.16, 1.58, 1.87 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.27, 1.69 [23:05:52] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - FAIL: [35072, 0, 0, 0, 0, 0, 0, 0, 0] in 349s [23:05:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:21] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 9.24, 9.97 [23:09:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.48, 1.76 [23:10:20] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:28] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:11:09] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-5066-dependabot/submodules/REL1_37/skins/chameleon-4b7531c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7344e9381c50 [23:11:11] [02miraheze/mediawiki] 07Universal-Omega 037344e93 - Revert "Bump skins/chameleon from `a923c46` to `4b7531c` (#5066)" [23:11:12] [02mediawiki] 07Universal-Omega opened pull request 03#5130: Revert "Bump skins/chameleon from `a923c46` to `4b7531c`" - 13https://github.com/miraheze/mediawiki/pull/5130 [23:11:14] [02mediawiki] 07Universal-Omega created branch 03revert-5066-dependabot/submodules/REL1_37/skins/chameleon-4b7531c - 13https://github.com/miraheze/mediawiki [23:11:15] [02mediawiki] 07Universal-Omega closed pull request 03#5130: Revert "Bump skins/chameleon from `a923c46` to `4b7531c`" - 13https://github.com/miraheze/mediawiki/pull/5130 [23:11:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6ce630e3f10e...3e3586b6a144 [23:11:18] [02miraheze/mediawiki] 07Universal-Omega 033e3586b - Revert "Bump skins/chameleon from `a923c46` to `4b7531c` (#5066)" (#5130) [23:11:20] [02mediawiki] 07Universal-Omega deleted branch 03revert-5066-dependabot/submodules/REL1_37/skins/chameleon-4b7531c - 13https://github.com/miraheze/mediawiki [23:11:21] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03revert-5066-dependabot/submodules/REL1_37/skins/chameleon-4b7531c [23:11:52] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.512 second response time [23:11:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.92, 10.22, 10.19 [23:12:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.50, 1.70 [23:13:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.810 second response time [23:13:52] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.888 second response time [23:14:43] [02mediawiki] 07Universal-Omega reviewed pull request 03#5023 commit - 13https://github.com/miraheze/mediawiki/pull/5023#discussion_r795265830 [23:15:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.15, 8.56, 8.24 [23:15:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.22, 9.04, 8.06 [23:16:12] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 969s [23:16:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.05, 8.50, 8.30 [23:19:10] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.77, 1.77 [23:19:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.62, 7.78, 7.79 [23:22:18] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.53, 1.68 [23:44:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.41, 9.47, 8.46 [23:47:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.14, 8.59, 8.31 [23:49:49] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.12, 5.70, 7.56 [23:50:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.57, 8.24, 8.22 [23:52:49] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.76, 4.43, 6.73 [23:53:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 9.24, 8.38 [23:56:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.24, 1.82, 1.72 [23:57:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.32, 8.77, 8.52 [23:58:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.00, 8.91, 8.23 [23:59:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.31, 1.53