[00:00:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.18, 3.07, 3.43 [00:01:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 8.53, 8.08 [00:02:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 2.94, 3.34 [00:02:10] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.66, 8.38, 8.10 [00:05:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 3.85, 3.65 [00:05:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.53, 8.88, 8.52 [00:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.63, 9.70, 8.81 [00:07:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.53, 3.55 [00:08:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.72, 4.00 [00:08:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.06, 9.93, 8.99 [00:09:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.39, 7.98, 8.27 [00:10:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.87, 3.95, 4.05 [00:10:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 9.78, 9.06 [00:11:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.94, 3.31 [00:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:20:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.83, 9.40, 9.01 [00:20:10] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.11, 1.35, 1.97 [00:20:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 9.67, 9.21 [00:21:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.45, 8.61 [00:22:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.99, 3.58 [00:22:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 10.41, 9.52 [00:23:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 4.26, 3.65 [00:24:34] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.05, 1.65, 1.90 [00:26:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.19, 1.46, 1.80 [00:26:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.55, 3.15, 3.37 [00:26:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 9.56, 9.43 [00:27:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.73, 3.56 [00:29:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.35, 4.15, 3.72 [00:30:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.21, 1.29, 1.64 [00:31:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.92, 3.68 [00:32:10] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:32:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.51, 3.46 [00:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 10.00, 9.48 [00:36:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 3.15, 3.33 [00:37:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 9.37, 8.88 [00:37:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.12, 3.40 [00:38:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.21, 3.15, 3.99 [00:38:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.80, 9.00, 9.26 [00:39:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.08, 8.65, 8.69 [00:43:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.43, 3.46 [00:44:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 3.86, 3.96 [00:44:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.51, 1.31 [00:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:45:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.69, 9.28, 8.89 [00:45:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.25, 3.40 [00:46:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.43, 1.30 [00:47:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.50, 9.09, 8.86 [00:47:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:48:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.82, 3.94 [00:48:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.07, 8.90, 9.00 [00:50:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 4.07, 4.00 [00:50:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.41, 8.57, 8.88 [00:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:52:34] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.54, 1.88, 1.57 [00:54:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.64, 3.85 [00:54:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.92, 1.87, 1.61 [00:54:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.87, 9.53, 9.19 [00:55:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 9.07, 8.88 [00:58:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:58:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.61, 1.70, 1.59 [00:58:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 9.86, 9.39 [01:04:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.01, 10.49, 9.67 [01:09:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.18, 9.82, 9.73 [01:09:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.09, 3.40 [01:09:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.79, 9.71 [01:10:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.61, 9.53 [01:11:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.73, 9.43, 9.59 [01:12:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.51, 8.25 [01:12:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.63, 10.21, 9.77 [01:14:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 9.58, 8.43 [01:16:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.49, 9.18, 9.45 [01:18:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.59, 9.65, 8.68 [01:18:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.51, 9.53 [01:19:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.95, 10.14, 9.69 [01:20:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 8.90, 8.52 [01:21:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.67, 2.18, 1.58 [01:22:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 8.32, 8.35 [01:23:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.63, 1.67, 1.46 [01:26:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.47, 3.05 [01:26:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.30, 8.62, 8.30 [01:27:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.36, 2.94 [01:28:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.64, 3.18 [01:28:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 8.43, 8.26 [01:29:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 2.92, 2.82 [01:32:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.49, 3.20 [01:32:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 9.01, 8.50 [01:33:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.16, 9.56, 8.83 [01:34:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.35, 3.17 [01:35:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.72, 9.33, 8.82 [01:37:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.46, 9.56, 8.96 [01:38:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.00, 3.08 [01:38:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 9.92, 9.03 [01:39:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.85, 9.36, 8.97 [01:40:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.75, 9.51, 8.97 [01:42:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.48, 9.64, 10.00 [01:44:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 3.51, 3.16 [01:46:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.55, 3.21 [01:46:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.21, 9.83, 9.95 [01:48:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.86, 9.71, 9.22 [01:48:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.77, 3.33 [01:50:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.61, 3.32 [01:52:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 4.09, 3.54 [01:52:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 9.43, 9.19 [01:52:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.37, 3.18 [01:54:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.54, 3.41 [01:54:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 3.05, 3.07 [01:56:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.04, 3.60 [01:58:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.59, 3.29 [01:58:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.72, 3.53 [01:58:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.39, 8.00, 8.47 [02:02:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.00, 3.27, 3.24 [02:02:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.23, 9.03 [02:02:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 8.78, 8.73 [02:04:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.25, 3.30, 3.37 [02:04:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.97, 8.17, 8.49 [02:05:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.62, 9.08, 10.00 [02:06:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.55, 9.47, 9.23 [02:07:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.28, 1.99, 1.47 [02:08:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.48, 3.47 [02:08:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.50, 4.24, 2.70 [02:08:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 9.07, 8.84 [02:09:27] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.68, 2.25, 1.62 [02:10:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.90, 9.33, 8.94 [02:11:21] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.79, 1.52 [02:12:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.01, 3.29 [02:12:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.52, 9.07 [02:13:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.81, 1.50, 1.45 [02:15:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 9.68, 9.84 [02:16:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.59, 3.45 [02:16:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.86, 9.56, 9.25 [02:18:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.69, 3.50 [02:20:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.67, 1.95 [02:20:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.89, 3.34 [02:22:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.33, 3.40 [02:22:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 9.67, 9.44 [02:22:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.71, 3.34 [02:24:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.80, 3.41 [02:26:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.52, 4.31, 3.76 [02:26:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 1.19, 1.65 [02:26:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.42, 3.32 [02:27:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.36, 9.43, 9.93 [02:28:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.81, 3.64 [02:28:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.20, 3.99, 3.53 [02:29:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.47, 3.25 [02:30:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 9.24, 8.95 [02:30:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.47, 3.39 [02:31:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.07, 10.15, 10.10 [02:31:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 3.19, 3.18 [02:32:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 3.05, 3.38 [02:32:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 8.97, 8.88 [02:32:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.85, 2.95, 3.21 [02:32:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.94, 9.58, 9.96 [02:34:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.20, 10.25, 9.74 [02:34:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.30, 9.94, 10.03 [02:36:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 9.64, 9.57 [02:36:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.34, 9.15, 9.75 [02:37:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.79, 9.22, 9.77 [02:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:38:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.76, 9.38, 9.74 [02:42:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.90, 9.78, 9.09 [02:42:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.11, 9.28, 9.67 [02:43:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.73, 1.57, 1.34 [02:44:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:44:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.41, 9.31 [02:45:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.32, 1.46, 1.33 [02:46:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.67, 3.34 [02:46:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.40, 9.09, 9.21 [02:48:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.35, 3.27 [02:48:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 9.57, 9.29 [02:48:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.83, 9.71, 9.59 [02:49:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:50:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.79, 9.10, 9.40 [02:52:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.55, 3.38 [02:52:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:52:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.59, 9.68, 9.57 [02:56:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.99, 9.51, 9.56 [02:57:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.57, 9.40, 9.15 [02:57:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.01, 7.73, 8.48 [02:58:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 3.03, 3.28 [02:59:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.16, 8.86, 9.01 [03:00:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.39, 7.73, 8.39 [03:00:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 4.00, 3.19 [03:01:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.93, 3.27 [03:02:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.23, 4.06, 3.70 [03:02:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.43, 3.07 [03:03:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.62, 3.23 [03:03:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.41, 9.76, 9.33 [03:04:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.75, 3.62 [03:04:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.14, 3.01 [03:04:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.46, 9.41 [03:05:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 3.30, 3.17 [03:05:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.74, 8.47, 8.50 [03:06:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 4.02, 3.74 [03:07:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 9.24, 8.78 [03:08:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.99, 3.75 [03:09:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 4.09, 3.52 [03:09:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.70, 9.41, 8.91 [03:10:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.87, 4.12, 3.84 [03:10:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.55, 9.05, 9.30 [03:11:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.66, 3.44 [03:11:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.65, 9.05 [03:12:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.20, 3.46, 3.63 [03:13:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 3.06, 3.25 [03:13:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 9.41, 9.04 [03:13:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.31, 9.58, 9.24 [03:14:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.73, 7.47, 7.03 [03:15:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.27, 9.81, 9.81 [03:17:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.08, 10.02, 9.86 [03:18:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.64, 3.61 [03:18:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.12, 6.94, 6.93 [03:22:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.39, 3.52 [03:22:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.33, 3.02 [03:22:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 9.54, 9.05 [03:24:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 2.95, 2.91 [03:26:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.34, 3.22, 3.40 [03:28:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.36, 8.21, 8.49 [03:28:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.54, 9.06, 9.17 [03:29:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 8.71, 8.77 [03:31:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.68, 8.26, 8.60 [03:32:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.94, 9.39, 8.83 [03:33:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.66, 9.21, 9.92 [03:34:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.94, 9.60, 8.98 [03:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 9.73, 9.28 [03:35:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.35, 7.85, 8.37 [03:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:38:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.37, 9.26, 9.21 [03:39:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.73, 1.26 [03:39:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 8.66, 8.63 [03:39:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.49, 9.46, 9.73 [03:40:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:41:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.93, 1.34, 1.17 [03:41:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.64, 9.46, 8.93 [03:41:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 9.23, 9.62 [03:44:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 4.13, 3.61 [03:45:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 9.43, 9.04 [03:46:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.95, 8.12, 8.46 [03:49:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.08, 9.46, 9.49 [03:51:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.51, 4.26, 3.38 [03:51:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.34, 4.14, 3.37 [03:51:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 9.03, 9.32 [03:53:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.26, 9.08, 8.73 [03:55:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.84, 8.81, 8.69 [03:59:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 8.08, 8.45 [04:03:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.94, 3.80 [04:04:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.35, 9.46, 9.14 [04:05:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.91, 3.89 [04:06:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 9.44, 9.19 [04:15:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.52, 2.95, 3.38 [04:15:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 2.81, 3.32 [04:15:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 8.82, 8.45 [04:15:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.15, 7.45, 8.27 [04:18:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.54, 8.05, 8.47 [04:18:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.28, 7.46, 8.27 [04:19:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.94, 7.82, 8.14 [04:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:21:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 8.17, 8.27 [04:22:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.22, 3.87 [04:25:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.37, 8.47, 8.39 [04:26:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.22, 4.37, 4.09 [04:27:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.66, 3.45 [04:28:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 4.16, 3.68 [04:28:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:28:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.51, 9.60, 8.90 [04:29:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.55, 9.54, 8.86 [04:30:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.59, 3.69, 3.90 [04:30:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 9.09, 8.79 [04:31:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.80, 2.99, 3.25 [04:31:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.98, 3.32, 3.49 [04:33:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 9.57, 9.04 [04:33:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 2.96, 3.34 [04:35:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 9.24, 8.79 [04:35:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.77, 10.01, 9.26 [04:37:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.76, 9.40, 9.14 [04:38:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.33, 2.33, 3.20 [04:38:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.19, 7.75, 8.36 [04:41:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.63, 9.26 [04:42:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.07, 3.28 [04:42:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.89, 8.15, 8.44 [04:43:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 9.54, 9.29 [04:46:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.31, 3.36 [04:47:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.84, 10.38, 9.66 [04:52:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 9.54, 8.64 [04:53:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 8.51, 7.61 [04:54:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.93, 9.56, 8.75 [04:56:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.00, 9.29, 8.33 [04:58:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.80, 9.18, 8.39 [04:58:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.01, 9.33, 8.81 [05:01:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 9.57, 9.92 [05:02:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.81, 8.56, 8.50 [05:02:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.88, 9.69, 9.08 [05:04:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.79, 10.47, 9.07 [05:07:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 9.65, 8.80 [05:07:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 9.87, 9.84 [05:10:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.71, 10.13, 9.32 [05:10:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 9.95, 9.29 [05:12:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.61, 9.94, 9.36 [05:16:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.15, 10.10, 9.50 [05:16:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 10.77, 9.77 [05:18:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 7.82, 7.06 [05:20:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 9.57, 9.43 [05:20:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.42, 6.84, 6.80 [05:21:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.30, 9.54, 9.69 [05:24:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.99, 7.76, 8.42 [05:24:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.50, 8.88, 9.57 [05:28:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 9.75, 9.50 [05:33:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.87, 7.45, 8.41 [05:34:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.86, 8.76, 7.29 [05:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.57, 9.53 [05:36:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 9.63, 9.56 [05:37:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.47, 9.35, 9.94 [05:38:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 8.85, 8.42 [05:40:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.96, 8.60, 8.39 [05:41:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.43, 2.98 [05:42:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.83, 8.33, 8.32 [05:42:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 9.46, 9.43 [05:43:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.09, 2.91 [05:46:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.05, 10.48, 9.21 [05:46:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 9.92, 9.63 [05:47:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.53, 7.01, 8.50 [05:54:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.48, 9.40 [05:55:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.99, 8.34, 7.70 [05:56:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.85, 9.58, 9.46 [05:57:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.33, 9.12, 8.08 [05:59:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.29, 9.14, 8.21 [06:02:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.22, 9.26, 9.81 [06:07:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.56, 7.85, 9.81 [06:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:08:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.59, 6.46, 8.29 [06:12:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.99, 7.57, 8.34 [06:13:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.63, 8.45, 8.43 [06:16:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.27, 8.16, 9.61 [06:21:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.53, 7.28, 8.44 [06:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:24:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.44, 3.01 [06:24:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.79, 1.55, 1.14 [06:26:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.93, 2.88 [06:26:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 8.82, 7.93 [06:26:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.66, 1.63, 1.22 [06:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:30:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.33, 8.35, 7.93 [06:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:30:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.38, 7.49, 8.45 [06:32:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 8.96, 8.39 [06:33:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.29, 9.25, 8.40 [06:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.50, 9.56, 8.68 [06:35:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.45, 8.31, 8.16 [06:36:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 9.59, 8.79 [06:36:46] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:36:59] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:10] oh f [06:37:20] Please tell me that's normal and not a slow enwp reaction [06:37:23] We fixed them [06:37:25] Cc CosmicAlpha [06:37:34] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:48] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:11] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:45] CosmicAlpha: you got access [06:38:45] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.674 second response time [06:38:56] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.901 second response time [06:39:22] RhinosF1: not right now, am away for tonight already. [06:39:32] Ok [06:39:34] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.018 second response time [06:39:37] Recovery though. [06:39:41] If things slow blame instant commons CosmicAlpha [06:39:46] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.636 second response time [06:40:07] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.019 second response time [06:40:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.05, 9.54, 8.89 [06:41:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 8.46, 8.17 [06:45:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.91, 8.50, 8.28 [06:48:34] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.25, 1.67, 1.42 [06:49:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.98, 9.35, 8.67 [06:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.92, 1.72, 1.46 [06:52:34] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.90, 2.10, 1.64 [06:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:55:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.67, 9.47, 8.89 [06:56:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.60, 1.86, 1.64 [06:59:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 9.81, 9.21 [06:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:00:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.19, 1.55, 1.57 [07:11:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.33, 9.25, 7.99 [07:13:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 9.51, 9.94 [07:13:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.29, 8.17 [07:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:15:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.23, 7.89, 8.43 [07:19:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.62, 8.02, 7.98 [07:21:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 9.34, 8.95 [07:21:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 8.88, 8.22 [07:22:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.62, 3.03 [07:22:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.27, 2.58 [07:23:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.42, 9.46, 8.51 [07:24:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 2.97, 2.56 [07:25:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.61, 7.79, 8.41 [07:25:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 9.80, 8.76 [07:26:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.56, 3.19 [07:28:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.38, 3.17 [07:29:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.86, 8.27, 8.39 [07:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:32:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.77, 7.10, 8.45 [08:06:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 8.58, 6.55 [08:07:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.66, 8.16, 6.80 [08:07:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:09:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.70, 8.71, 7.27 [08:09:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 8.61, 7.72 [08:11:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 8.60, 7.82 [08:11:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 9.12, 7.48 [08:12:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:12:35] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [08:12:35] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:13:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.67, 8.95, 7.62 [08:13:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.65, 9.13, 7.82 [08:17:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.13, 8.50, 8.04 [08:17:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:20:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 8.85, 8.28 [08:23:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 9.97, 8.83 [08:26:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.36, 9.53, 8.67 [08:27:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.29, 8.49 [08:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:32:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 9.67, 9.08 [08:33:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.72, 9.09, 8.75 [08:33:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 9.97, 9.46 [08:34:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.21, 9.81, 9.79 [08:35:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.35, 7.75, 8.30 [08:35:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:40:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.63, 7.32, 8.30 [08:40:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.15, 6.42, 8.31 [08:42:22] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:44:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 8.08, 8.39 [08:46:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.04, 7.76, 8.21 [08:47:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.22, 7.81, 8.47 [08:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:48:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.49, 8.12, 7.40 [08:50:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.32, 7.92, 7.42 [08:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:14:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:14:04] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:47:45] [02mw-config] 07ugochimobi opened pull request 03#4532: Update Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4532 [09:48:48] miraheze/mw-config - ugochimobi the build passed. [09:50:30] [02mw-config] 07ugochimobi synchronize pull request 03#4532: Update Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4532 [09:51:30] miraheze/mw-config - ugochimobi the build passed. [09:53:38] [02mw-config] 07ugochimobi edited pull request 03#4532: Update Wikibase.php - 13https://github.com/miraheze/mw-config/pull/4532 [09:59:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:08:03] [02mw-config] 07ugochimobi opened pull request 03#4533: WBQuality constraint configs for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4533 [10:09:05] miraheze/mw-config - ugochimobi the build passed. [10:13:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:19:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.87, 8.39, 7.81 [10:21:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.08, 7.68, 7.62 [10:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:38:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:45:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [10:45:19] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:48:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:00:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:05:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:11:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 8.50, 7.65 [11:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:15:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.45, 7.98, 7.69 [11:19:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:43:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:47:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 8.41, 7.64 [11:53:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.51, 8.31, 7.91 [11:53:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:56:01] [02mw-config] 07ugochimobi opened pull request 03#4534: GeoData config for gratisdata - 13https://github.com/miraheze/mw-config/pull/4534 [11:57:03] miraheze/mw-config - ugochimobi the build passed. [12:08:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.40, 8.40, 7.83 [12:11:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 9.71, 8.37 [12:13:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.91, 8.75, 7.89 [12:14:38] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:15:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.35, 8.48, 7.77 [12:15:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 9.31, 8.18 [12:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.48, 2.96 [12:16:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.86, 9.18, 8.59 [12:16:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.10, 8.37, 7.82 [12:17:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.61, 8.70, 8.10 [12:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:20:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.56, 8.13, 8.31 [12:21:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.67, 7.44, 7.76 [12:22:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.13, 2.96 [12:24:34] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.10, 1.85, 1.39 [12:26:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 8.64, 8.31 [12:26:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.63, 1.74, 1.40 [12:26:53] [02mw-config] 07ugochimobi opened pull request 03#4535: Set globe for coordinates added to GeoData - 13https://github.com/miraheze/mw-config/pull/4535 [12:28:01] miraheze/mw-config - ugochimobi the build passed. [12:28:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.48, 1.61, 1.39 [12:29:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.50, 8.40, 8.37 [12:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:34:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.67, 3.26 [12:34:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.43, 2.91 [12:35:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.90, 8.62 [12:35:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.08, 9.72, 8.82 [12:36:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.27, 3.17 [12:36:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 4.02, 3.20 [12:36:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.16, 9.17, 8.41 [12:37:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 9.37, 8.57 [12:38:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.58, 3.14 [12:38:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.14, 9.13, 8.42 [12:39:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.82, 9.58, 8.97 [12:40:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 3.31, 3.10 [12:40:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 9.37, 8.59 [12:41:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.85, 8.45, 8.37 [12:42:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.71, 8.80 [12:42:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 9.50, 8.85 [12:44:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 9.37, 8.80 [12:44:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.24, 8.81, 8.58 [12:48:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.94, 3.50 [12:48:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.65, 8.27, 8.48 [12:48:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.13, 7.99, 8.36 [12:48:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.87, 8.21, 8.47 [12:49:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.67, 9.30, 8.97 [12:51:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.38, 9.05 [12:52:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 9.56, 8.96 [12:53:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.67, 10.14, 9.36 [12:54:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.14, 8.55, 8.50 [12:55:10] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.42, 1.95, 1.66 [12:55:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 9.57, 9.26 [12:56:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.81, 3.03, 3.27 [12:56:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.41, 8.85, 8.83 [12:57:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.61, 1.79, 1.63 [12:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:58:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.27, 8.18, 8.38 [13:01:03] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.64, 1.66, 1.62 [13:02:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 9.86, 9.11 [13:05:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.58, 7.86, 8.41 [13:06:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 9.01, 8.95 [13:10:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.01, 7.67, 8.42 [13:13:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:14:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.12, 1.91, 1.42 [13:14:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.68, 3.26 [13:16:42] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.94, 1.58, 1.36 [13:16:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.70, 3.31 [13:17:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [13:17:05] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:18:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.95, 3.22 [13:20:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.61, 8.94, 8.55 [13:22:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.33, 3.12 [13:23:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:24:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 2.77, 3.07 [13:28:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 8.49, 8.16 [13:30:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 8.18, 8.08 [13:31:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.48, 9.68, 8.95 [13:33:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.74, 3.39 [13:33:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.76, 3.31 [13:33:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.68, 3.21 [13:33:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.21, 8.86 [13:35:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.78, 9.53, 8.31 [13:35:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 8.97, 8.40 [13:36:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.44, 4.33, 3.66 [13:37:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.85, 10.09, 9.30 [13:37:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 9.33, 8.40 [13:37:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.85, 8.33, 8.22 [13:39:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.19, 9.45, 9.16 [13:39:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.12, 9.69, 8.65 [13:39:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 8.93, 8.45 [13:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.58, 3.48 [13:41:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.39, 3.34 [13:41:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 9.61, 8.76 [13:41:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.48, 8.70 [13:42:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.89, 3.69 [13:43:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.21, 3.36 [13:43:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 9.00, 8.62 [13:43:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.91, 9.67, 8.87 [13:44:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 4.47, 3.93 [13:45:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.43, 10.04, 9.45 [13:46:51] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:47:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.18, 8.74 [13:49:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.10, 9.88, 9.56 [13:50:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.46, 3.69 [13:51:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 9.90, 9.55 [13:51:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 10.20, 9.71 [13:52:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 4.03, 3.87 [13:53:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.75, 8.94, 8.91 [13:53:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.05, 9.92, 9.29 [13:55:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.28, 7.75, 8.48 [13:55:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 9.68, 9.27 [13:55:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.43, 9.58, 9.60 [13:57:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.27, 3.21 [13:57:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.87, 9.89, 9.60 [13:57:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.34, 9.87 [13:58:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.74, 3.45 [13:59:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 3.70, 3.37 [13:59:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 10.04, 9.49 [13:59:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.61, 9.53 [14:00:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.42, 3.36 [14:01:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.33, 9.93, 9.66 [14:01:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.24, 9.97, 9.84 [14:02:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.13, 3.27 [14:03:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.56, 3.41 [14:03:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.92, 8.87, 9.31 [14:03:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.34, 9.10, 8.02 [14:07:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.64, 9.23, 8.84 [14:08:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.67, 3.96 [14:10:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.78, 3.96 [14:11:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.87, 3.62 [14:12:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.45, 3.82 [14:12:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.00, 9.72, 9.88 [14:14:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 10.04, 9.99 [14:17:15] [02miraheze/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/rectalogic/notify-irc-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/73f2b6e87202 [14:17:17] [02miraheze/mw-config] 07dependabot[bot] 0373f2b6e - Bump rectalogic/notify-irc from 1 to 2 [14:17:18] [02mw-config] 07dependabot[bot] opened pull request 03#4536: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/mw-config/pull/4536 [14:17:20] [02mw-config] 07dependabot[bot] labeled pull request 03#4536: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/mw-config/pull/4536 [14:17:21] [02mw-config] 07dependabot[bot] labeled pull request 03#4536: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/mw-config/pull/4536 [14:17:23] [02mw-config] 07dependabot[bot] created branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/mw-config [14:18:49] miraheze/mw-config - dependabot[bot] the build passed. [14:18:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.55, 9.72, 9.90 [14:20:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 8.71, 8.64 [14:21:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.92, 3.92 [14:22:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 9.26, 8.88 [14:25:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.65, 7.31, 8.22 [14:26:10] [02mw-config] 07RhinosF1 closed pull request 03#4536: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/mw-config/pull/4536 [14:26:11] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2475ebc96642...d03ae7297eba [14:26:13] [02miraheze/mw-config] 07dependabot[bot] 03d03ae72 - Bump rectalogic/notify-irc from 1 to 2 (#4536) [14:26:14] [02mw-config] 07RhinosF1 deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/mw-config [14:26:16] [02miraheze/mw-config] 07RhinosF1 deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 [14:27:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 4.22, 4.04 [14:27:19] miraheze/mw-config - RhinosF1 the build passed. [14:27:20] [02mw-config] 07RhinosF1 commented on pull request 03#4534: GeoData config for gratisdata - 13https://github.com/miraheze/mw-config/pull/4534#issuecomment-1081941618 [14:27:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.78, 3.37, 2.96 [14:30:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 3.94, 3.72 [14:31:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.07, 7.11, 8.20 [14:31:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.33, 3.07 [14:32:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.67, 3.64 [14:33:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.17, 8.73, 9.77 [14:33:24] RECOVERY - bast121 APT on bast121 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:33:33] RECOVERY - bast101 APT on bast101 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [14:33:39] RECOVERY - es101 APT on es101 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:33:52] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:33:53] RECOVERY - phab121 APT on phab121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:33:53] RECOVERY - db101 APT on db101 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [14:33:57] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [14:33:59] RECOVERY - es121 APT on es121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:34:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 4.08, 3.80 [14:34:06] RECOVERY - ns2 APT on ns2 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:34:12] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:34:14] RECOVERY - es111 APT on es111 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:34:17] RECOVERY - mw122 APT on mw122 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [14:34:20] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:34:20] RECOVERY - mw101 APT on mw101 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [14:34:21] RECOVERY - db121 APT on db121 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [14:34:21] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:34:25] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [14:34:26] RECOVERY - mw121 APT on mw121 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [14:34:28] RECOVERY - mail121 APT on mail121 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [14:34:28] RECOVERY - mw102 APT on mw102 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [14:34:29] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [14:34:30] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:34:32] RECOVERY - test101 APT on test101 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [14:34:35] RECOVERY - mon111 APT on mon111 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [14:34:35] RECOVERY - db111 APT on db111 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [14:34:39] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [14:34:45] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:34:46] RECOVERY - mw112 APT on mw112 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [14:34:51] RECOVERY - db112 APT on db112 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [14:35:02] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 50 packages available for upgrade (0 critical updates). [14:35:04] RECOVERY - mem121 APT on mem121 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:35:10] RECOVERY - matomo101 APT on matomo101 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [14:35:12] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:35:19] RECOVERY - ns1 APT on ns1 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [14:35:21] RECOVERY - mw111 APT on mw111 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [14:35:22] RECOVERY - mem101 APT on mem101 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [14:36:59] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:12] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.77, 3.76 [14:38:46] RECOVERY - cp31 APT on cp31 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [14:38:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.90, 7.15, 8.35 [14:38:55] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.319 second response time [14:39:11] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.020 second response time [14:39:12] RECOVERY - cp30 APT on cp30 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [14:39:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 3.22, 5.98, 8.24 [14:39:52] RECOVERY - cp21 APT on cp21 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [14:40:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.67, 2.69, 3.37 [14:40:21] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [14:40:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 3.65, 6.77, 7.96 [14:40:55] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [14:41:02] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [14:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.25, 2.28, 3.59 [14:41:06] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 98% [14:41:06] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.019 second response time [14:41:07] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [14:41:07] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.355 second response time [14:41:15] PROBLEM - cp30 Varnish Backends on cp30 is WARNING: No backends detected. If this is an error, see readme.txt [14:41:19] PROBLEM - cp20 Varnish Backends on cp20 is WARNING: No backends detected. If this is an error, see readme.txt [14:41:51] PROBLEM - cp21 Varnish Backends on cp21 is WARNING: No backends detected. If this is an error, see readme.txt [14:42:04] PROBLEM - cp31 Varnish Backends on cp31 is WARNING: No backends detected. If this is an error, see readme.txt [14:42:05] PROBLEM - cp31 HTTPS on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.331 second response time [14:42:09] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [14:42:13] JohnLewis, paladox, Reception123: icinga, mw, grafana down [14:42:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:42:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:42:36] eek [14:43:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.03, 1.52, 3.15 [14:43:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:44:26] no traffic flowing cp* -> mw101 [14:44:42] I should be around in an hour or so but either way our best bet is definitely someone from Infra [14:45:22] hmm [14:46:30] weird [14:46:50] paladox: mw traffic drops off a cliff edge [14:46:56] it's something front end [14:47:38] https://www.irccloud.com/pastebin/3IuzHItv/ [14:47:39] !log [@test101] starting deploy of {'config': True} to all [14:47:40] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [14:47:46] i'll reboot [14:48:00] !log reboot cp20 & cp21 [14:48:38] hmm it's still down and i dunno why [14:48:54] paladox: I can ping cp20 fine [14:48:58] from mw101 [14:49:02] yes cp20/21 are up [14:49:05] i'm ssh'd in [14:49:16] and if you get an nginx error screen it means you connected to it [14:49:46] paladox: if connection is refused it's something between mw* and cp* [14:50:04] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [14:50:05] RECOVERY - cp31 HTTPS on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3210 bytes in 0.412 second response time [14:50:27] i don't see it starting port 81 [14:50:37] it's varnish port 81 isn't starting [14:51:02] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 2% [14:53:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.37, 1.73, 0.84 [14:53:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.98, 3.72, 3.04 [14:53:10] i don't see any errors in varnish log (journalctl) [14:53:18] !log [@mwtask111] starting deploy of {'config': True} to all [14:53:22] paladox: cp31 is fine [14:53:24] no MirahezeLSBot [14:53:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.48, 5.26, 3.22 [14:53:46] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 28s [14:53:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:53:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:54:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.80, 6.18, 3.93 [14:54:24] paladox: what about syslog? [14:54:57] that would be impossible to go through quickly because it's in graylog [14:56:05] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw101 mw102 [14:57:06] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3209 bytes in 0.020 second response time [14:57:12] ok it now works on cp20 [14:57:18] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [14:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:57:51] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [14:58:20] that was weird (and i did nothing apart from try restarting varnish again) [14:58:30] I'm getting persistent 503s on gratispaideiawiki [14:58:32] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 3% [14:58:44] 502s too [14:58:49] ugochimobi: see the conversation [14:58:58] we're middle of an incident [14:59:03] oh [14:59:03] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:07] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3210 bytes in 0.156 second response time [14:59:14] icinga alert blinded me [14:59:21] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.231 second response time [14:59:31] ugochimobi: the alert saying everything was down? [14:59:49] no [14:59:49] RhinosF1: seems it's now resolved now varnish both working on cp20/21 (and port 81 works) [14:59:51] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:51] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:51] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [15:00:05] I mean the alerts are everywhere [15:00:09] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 6% [15:00:14] paladox: it's not 100% [15:00:14] but gratispaideiawiki seems back [15:00:21] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:23] i think some stuff are still laggy [15:00:32] ugochimobi: please stop talking [15:00:34] yeh but that's been like that for years [15:00:52] paladox: icinga is still too slow to use [15:01:00] icinga fast for me [15:01:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.06, 1.97, 1.51 [15:01:13] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.669 second response time [15:01:20] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.320 second response time [15:01:45] paladox: still a fair few alerts now I'm in but looks on the way [15:01:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.022 second response time [15:01:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.366 second response time [15:01:54] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:08] it should balance out once it notices is everything is okay [15:02:17] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:18] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [15:03:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.68, 1.52, 1.40 [15:03:26] paladox: cp30 still isn't happy [15:03:54] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.253 second response time [15:04:04] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [15:04:19] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.449 second response time [15:04:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:04:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:04:33] restarted varnish on cp30 should work now [15:04:50] paladox: yep [15:04:54] !log restarted varnish on cp20/21/30 [15:04:55] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 1% [15:04:57] not sure why mw122 was down [15:05:07] RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3210 bytes in 0.425 second response time [15:05:11] when i logged in it recovered [15:05:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:05:15] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [15:07:35] paladox: everything back to normal states [15:07:44] ok, thanks! [15:08:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.84, 8.67, 6.76 [15:13:05] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.82, 7.97, 7.16 [15:15:08] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.024 second response time [15:18:22] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:18:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.99, 9.00, 7.72 [15:20:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22337 bytes in 4.020 second response time [15:20:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.23, 8.92, 6.87 [15:21:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.59, 8.84, 7.54 [15:22:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.41, 9.73, 8.38 [15:23:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.55, 9.75, 8.05 [15:24:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.99, 8.63 [15:24:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 9.60, 7.63 [15:25:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.91, 7.95, 6.46 [15:26:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.69, 9.52, 8.63 [15:27:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.55, 7.23, 6.37 [15:27:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.73, 9.52, 8.34 [15:28:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 9.63, 8.39 [15:28:46] PROBLEM - data.memipedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.memipedia.ga' expires in 15 day(s) (Thu 14 Apr 2022 15:13:58 GMT +0000). [15:30:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.27, 8.06 [15:31:06] [02mw-config] 07ugochimobi commented on pull request 03#4534: GeoData config for gratisdata - 13https://github.com/miraheze/mw-config/pull/4534#issuecomment-1082025854 [15:31:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.29, 8.51, 6.83 [15:32:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.83, 8.84, 8.06 [15:33:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fc2e70c83edf...4dacd31ded3c [15:33:05] [02miraheze/ssl] 07MirahezeSSLBot 034dacd31 - Bot: Update SSL cert for data.memipedia.ga [15:33:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.07, 8.06, 6.88 [15:34:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.76, 9.41, 8.88 [15:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.34, 9.76, 8.48 [15:36:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.43, 8.81, 8.74 [15:36:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.33, 8.85, 8.31 [15:37:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.41, 8.48, 8.40 [15:40:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.21, 8.38, 8.28 [15:42:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.38, 9.72, 9.11 [15:44:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.68, 9.06, 8.95 [15:44:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 8.99, 8.66 [15:44:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 8.92, 8.54 [15:51:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 9.28, 8.84 [15:52:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 9.02, 8.85 [15:53:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 9.32, 8.91 [15:54:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 9.77, 9.86 [15:57:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 9.14, 9.00 [15:58:31] RECOVERY - data.memipedia.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'data.memipedia.ga' will expire on Mon 27 Jun 2022 14:32:55 GMT +0000. [16:01:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.54, 8.92, 8.80 [16:04:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 10.11, 9.82 [16:04:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.73, 8.20, 8.49 [16:05:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.16, 8.94 [16:10:07] paladox: any idea why a dump would fail? see https://scruffy.miraheze.org/wiki/Special:DataDump [16:10:10] [url] Permission error - Scruffy Wiki | scruffy.miraheze.org [16:10:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 9.86, 9.87 [16:10:47] Nope, you'll need to either check the logs or to try and regenerate it [16:10:55] it could be a large wiki [16:11:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.07, 9.62, 9.17 [16:12:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.45, 9.87, 9.85 [16:12:46] I did try to regenerate [16:12:58] and hmm, maybe it's because it's large yeah [16:15:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.16, 8.52, 8.48 [16:17:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.16, 9.57, 9.18 [16:17:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.61, 8.77, 9.06 [16:19:16] doesn't seem to be particularly large [16:19:28] thats what she said [16:19:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 9.25, 8.75 [16:23:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.49, 9.80, 9.11 [16:25:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.85, 9.98, 9.72 [16:27:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 8.74, 8.74 [16:27:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.23, 9.98, 9.28 [16:29:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.84, 8.47, 8.65 [16:30:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.50, 9.24, 9.82 [16:31:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.08, 8.00, 8.46 [16:32:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.36, 9.89, 9.97 [16:33:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.15, 9.88, 9.49 [16:35:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.70, 10.12, 9.62 [16:37:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 9.83, 9.60 [16:37:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.59, 9.19, 8.84 [16:38:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.11, 9.39, 9.85 [16:39:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 10.11, 9.73 [16:41:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.78, 7.85, 8.47 [16:44:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.44, 9.66 [16:45:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.24, 9.13, 9.56 [16:47:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.99, 7.58, 8.29 [16:48:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.38, 9.48, 9.68 [16:49:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.86, 9.23, 8.81 [16:50:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 9.64, 9.69 [16:51:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.72, 9.53, 9.54 [16:53:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.69, 9.10, 9.40 [16:53:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.91, 9.18, 8.95 [16:54:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 9.35, 9.54 [16:56:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.55, 9.58 [16:58:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 9.27, 9.48 [16:59:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.18, 9.78, 9.57 [17:01:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.64, 9.54 [17:02:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.65, 9.64, 9.55 [17:06:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 9.67, 9.63 [17:09:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.77, 7.89, 8.48 [17:11:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.49, 9.84, 9.61 [17:12:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.14, 9.29, 9.42 [17:12:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.60, 3.95 [17:13:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.93, 8.33, 8.60 [17:14:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.69, 3.93 [17:15:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.83, 8.17, 8.50 [17:16:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 9.23, 9.42 [17:18:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.64, 9.97, 9.66 [17:19:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.24, 9.21, 8.87 [17:20:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 9.71, 9.60 [17:20:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.86, 3.96 [17:21:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 9.92, 9.90 [17:22:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.95, 9.81, 9.62 [17:23:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.00, 10.23, 10.01 [17:26:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.51, 9.58 [17:28:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.10, 8.56 [17:28:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.23, 10.01, 9.74 [17:29:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.96, 9.21, 9.72 [17:30:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.03, 8.90, 8.54 [17:30:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 9.67, 9.67 [17:31:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.94, 8.01, 8.50 [17:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:32:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.93, 9.77 [17:32:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 3.66, 3.67 [17:34:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 9.56, 8.87 [17:36:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.77, 9.47, 9.72 [17:37:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:38:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.12, 9.28, 8.96 [17:39:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.49, 9.00, 9.23 [17:43:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.80, 8.77, 9.13 [17:45:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.00, 7.87, 8.44 [17:46:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.82, 3.93 [17:49:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.52, 9.64, 8.80 [17:51:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.81, 7.74, 8.50 [17:51:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.58, 9.36, 8.81 [17:52:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 9.25, 9.33 [17:52:38] [02mw-config] 07Universal-Omega reviewed pull request 03#4535 commit - 13https://github.com/miraheze/mw-config/pull/4535#discussion_r837750432 [17:52:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.06, 4.37, 4.11 [17:54:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.30, 8.74, 9.13 [17:56:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.45, 9.35 [17:57:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.01, 8.05, 8.46 [18:01:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 8.68, 8.58 [18:02:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.83, 9.58 [18:03:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.61, 8.22, 8.35 [18:04:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.45, 10.38, 9.81 [18:05:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.44, 9.84, 9.05 [18:05:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.13, 7.85, 8.20 [18:06:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.69, 3.98 [18:07:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.98, 9.20 [18:12:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.36, 3.91 [18:12:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 9.23, 8.71 [18:14:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.35, 3.74, 3.96 [18:16:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.07, 9.69, 9.01 [18:16:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.27, 9.41, 9.87 [18:16:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.50, 3.65 [18:18:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.53, 3.64 [18:19:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.47, 3.11, 3.99 [18:19:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 9.46, 9.09 [18:20:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 9.31, 9.71 [18:20:34] [02mw-config] 07ugochimobi reviewed pull request 03#4535 commit - 13https://github.com/miraheze/mw-config/pull/4535#discussion_r837778315 [18:20:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 9.15, 8.97 [18:21:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.63, 4.07 [18:22:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.68, 8.43, 9.35 [18:22:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.11, 4.71, 4.03 [18:22:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.63, 8.89, 8.90 [18:24:34] [02mw-config] 07ugochimobi synchronize pull request 03#4535: Set globe for coordinates added to GeoData - 13https://github.com/miraheze/mw-config/pull/4535 [18:25:37] miraheze/mw-config - ugochimobi the build passed. [18:25:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 7.89, 8.49 [18:27:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.60, 3.98 [18:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.69, 4.45, 4.24 [18:29:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.41, 8.37, 8.56 [18:30:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.77, 3.87 [18:30:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.42, 7.71, 8.39 [18:32:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:32:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.55, 8.03, 7.16 [18:33:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.70, 3.99 [18:34:54] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.69, 7.46, 7.05 [18:37:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 4.29, 4.16 [18:38:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.34, 9.27, 9.07 [18:39:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.36, 8.79, 8.60 [18:40:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 9.14, 9.05 [18:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.80, 3.99 [18:41:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.52, 10.16, 9.23 [18:42:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.68, 3.99 [18:42:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.23, 9.33, 8.29 [18:43:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.76, 8.24, 8.40 [18:44:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.71, 3.95 [18:45:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.39, 9.96, 8.83 [18:47:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.54, 11.00, 9.34 [18:50:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 2.98, 3.33 [18:51:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 8.93, 8.51 [18:52:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.61, 3.90 [18:53:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.72, 3.77 [18:53:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 9.64, 9.60 [18:54:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.69, 1.74, 1.48 [18:55:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 9.76, 9.46 [18:56:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.86, 3.91 [18:56:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.72, 3.97, 3.56 [18:58:31] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:00:30] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.522 second response time [19:01:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.86, 8.32, 8.46 [19:02:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 9.18, 8.98 [19:03:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.03, 9.81, 9.49 [19:04:37] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.29, 1.58, 1.57 [19:05:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.83, 9.20, 9.22 [19:05:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 9.36, 9.35 [19:06:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.37, 3.82 [19:06:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.63, 3.77 [19:06:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 8.99, 8.68 [19:07:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.31, 3.85 [19:07:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 9.31, 9.27 [19:09:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.88, 10.06, 9.54 [19:10:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.45, 9.70, 9.00 [19:12:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 3.38, 3.51 [19:13:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.52, 3.72 [19:13:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.41, 8.90, 7.88 [19:13:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 10.00, 9.70 [19:13:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.50, 9.32 [19:14:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.55, 3.66 [19:14:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.84, 9.65 [19:14:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.52, 3.00, 3.36 [19:15:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.37, 3.63 [19:15:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.29, 8.03, 7.70 [19:15:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.61, 10.04, 9.73 [19:16:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.61, 3.67 [19:16:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.29, 10.88, 10.08 [19:17:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.55, 9.39, 9.55 [19:17:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.70, 9.66, 9.42 [19:19:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.43, 3.56 [19:19:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 10.12, 9.62 [19:21:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.34, 3.51 [19:21:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.70, 9.27, 9.37 [19:22:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 9.62, 9.81 [19:25:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.74, 3.61 [19:25:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.91, 9.95, 9.64 [19:26:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.64, 3.61 [19:26:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.13, 9.78, 9.80 [19:27:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 9.42, 9.49 [19:27:59] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [19:27:59] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:28:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.85, 10.34, 10.00 [19:29:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 8.80, 9.01 [19:30:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.70, 3.65 [19:30:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.79, 9.81, 9.78 [19:31:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 8.94, 9.05 [19:32:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.94, 3.73 [19:33:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.51, 9.24 [19:36:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.61, 3.66 [19:37:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.99, 3.91 [19:38:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.71, 9.45, 9.74 [19:40:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 3.97, 3.75 [19:41:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.29, 4.15, 3.94 [19:41:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.74, 10.06, 9.65 [19:42:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.69, 3.67 [19:42:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.65, 8.53, 8.20 [19:42:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.65, 3.36 [19:43:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.84, 3.86 [19:44:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.79, 3.69 [19:44:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.46, 8.31, 8.16 [19:44:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.48, 3.34 [19:46:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.17, 3.25 [19:48:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.84, 3.76 [19:48:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.32, 9.24, 8.59 [19:48:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.83, 9.40, 9.42 [19:49:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.25, 9.56, 9.73 [19:50:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.92, 3.80 [19:50:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.55, 8.47, 8.40 [19:52:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.60, 3.69 [19:53:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.57, 3.42 [19:55:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.09, 3.38, 3.37 [19:56:00] [02mw-config] 07ugochimobi edited pull request 03#4534: GeoData config for gratisdata - 13https://github.com/miraheze/mw-config/pull/4534 [19:56:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 3.78, 3.71 [19:58:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 9.92, 9.85 [20:00:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 9.67, 9.99 [20:00:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.68, 10.16, 9.94 [20:01:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.19, 9.25 [20:01:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.91, 9.16, 9.70 [20:04:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.89, 3.86 [20:04:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 10.16, 10.10 [20:05:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 9.54, 9.38 [20:06:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 9.47, 9.83 [20:06:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.16, 9.84, 9.98 [20:07:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 10.67, 9.81 [20:07:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.41, 9.63 [20:08:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.57, 4.12, 3.92 [20:09:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.81, 3.63 [20:09:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.46, 9.13, 9.51 [20:10:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.83, 9.79, 9.85 [20:10:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.80, 8.39, 7.72 [20:11:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.25, 3.44 [20:12:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.98, 3.94 [20:12:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.46, 8.81, 8.26 [20:12:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 9.50, 9.74 [20:12:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 9.05, 8.04 [20:13:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.20, 3.11, 3.36 [20:14:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.67, 7.85, 7.96 [20:14:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.07, 8.88, 8.09 [20:17:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.89, 9.40, 9.73 [20:18:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.62, 8.06, 7.95 [20:19:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.35, 9.72, 9.43 [20:20:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.83, 3.84 [20:20:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.32, 3.13 [20:21:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.45, 3.75, 3.45 [20:22:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.68, 3.28 [20:22:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.77, 8.91, 9.18 [20:22:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 8.57, 8.86 [20:23:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.63, 3.44 [20:24:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.61, 3.31 [20:24:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 8.88, 8.95 [20:25:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.72, 3.48 [20:26:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.17, 4.72, 3.74 [20:26:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.26, 8.87, 9.15 [20:27:31] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [20:29:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.32, 3.56, 3.52 [20:29:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 10.00, 9.82 [20:31:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.57, 2.96, 3.30 [20:32:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.65, 3.98 [20:32:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.68, 3.63 [20:33:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.45, 9.27 [20:33:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.66, 10.10, 9.90 [20:34:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.40, 10.24, 9.55 [20:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.67, 9.68, 9.25 [20:35:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.59, 9.43, 9.70 [20:36:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 9.75, 9.35 [20:37:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.46, 9.52, 9.37 [20:39:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.01, 9.56, 9.39 [20:40:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.16, 3.40 [20:40:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.25, 9.85, 9.73 [20:41:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.30, 9.03, 9.39 [20:42:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.40, 10.33, 9.93 [20:43:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.97, 8.32, 9.09 [20:44:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.39, 3.47 [20:44:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.26, 9.51, 9.68 [20:45:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.71, 9.84, 9.64 [20:46:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.11, 3.35 [20:47:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.57, 10.14, 9.78 [20:47:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 9.25, 9.28 [20:49:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.05, 9.53, 9.61 [20:50:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.74, 3.64 [20:50:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.52, 3.49 [20:51:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.81, 9.96, 9.61 [20:52:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.32, 3.50 [20:53:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.30, 9.73, 9.59 [20:53:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 10.33, 9.80 [20:54:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.19, 9.20, 9.00 [20:55:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.40, 9.41, 9.51 [20:56:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.63, 3.58 [20:57:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.50, 10.38, 9.87 [20:57:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.87, 9.75 [20:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.87, 4.12, 3.71 [20:58:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.51, 3.55 [20:58:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 9.83, 9.34 [20:59:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.41, 3.14 [21:00:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.70, 3.60 [21:00:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.47, 3.51 [21:00:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 10.38, 9.71 [21:01:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.80, 3.33, 3.15 [21:01:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.57, 2.93, 3.33 [21:02:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.15, 3.20, 3.40 [21:02:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:03:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 10.00, 9.86 [21:04:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 9.94, 9.43 [21:05:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.71, 3.37 [21:05:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:05:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.33, 3.44 [21:06:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.72, 3.60 [21:07:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.91, 3.25, 3.39 [21:08:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.88, 9.21, 9.29 [21:09:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.91, 3.50 [21:10:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 3.72, 3.58 [21:10:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:10:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.30, 9.40, 9.33 [21:11:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.63, 3.51 [21:12:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.93, 3.68 [21:12:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 9.62, 9.71 [21:12:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.35, 9.10, 9.24 [21:13:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.72, 3.53 [21:13:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.12, 3.33 [21:14:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.83, 4.18, 3.83 [21:14:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 9.65, 9.70 [21:15:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.12, 5.06, 4.00 [21:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:17:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.47, 3.44 [21:18:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.06, 8.92, 9.43 [21:18:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.22, 9.49, 9.26 [21:19:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.53, 9.80, 9.26 [21:20:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.14, 9.63, 9.61 [21:24:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.00, 3.56, 3.80 [21:24:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 9.67, 9.65 [21:24:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.91, 9.28, 9.31 [21:25:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.86, 3.89 [21:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:25:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.23, 3.37 [21:26:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.71, 3.81 [21:27:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 4.45, 4.10 [21:28:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.89, 3.89 [21:29:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 9.55, 9.67 [21:29:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.56, 4.29, 3.78 [21:30:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.38, 4.07 [21:30:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 10.32, 9.92 [21:31:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:31:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.85, 3.68 [21:34:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.40, 3.75 [21:34:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.67, 8.93, 8.95 [21:35:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.05, 9.22, 8.97 [21:35:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.33, 9.03, 9.27 [21:36:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.76, 8.30, 8.74 [21:37:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.92, 9.52, 9.12 [21:37:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 8.88, 9.19 [21:38:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.60, 3.72 [21:38:19] [02miraheze/MirahezeMagic] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/rectalogic/notify-irc-2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/683bcc89da94 [21:38:21] [02miraheze/MirahezeMagic] 07dependabot[bot] 03683bcc8 - Bump rectalogic/notify-irc from 1 to 2 [21:38:22] [02MirahezeMagic] 07dependabot[bot] opened pull request 03#332: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/MirahezeMagic/pull/332 [21:38:24] [02MirahezeMagic] 07dependabot[bot] created branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/MirahezeMagic [21:38:25] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#332: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/MirahezeMagic/pull/332 [21:38:27] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#332: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/MirahezeMagic/pull/332 [21:39:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 2.96, 3.30 [21:40:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.56, 3.70 [21:40:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.21, 9.42, 9.84 [21:40:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.02, 7.92, 8.46 [21:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.46, 3.93 [21:41:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 9.10, 8.99 [21:42:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.65, 3.71 [21:42:48] [02MirahezeMagic] 07Universal-Omega closed pull request 03#332: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/MirahezeMagic/pull/332 [21:42:50] [02MirahezeMagic] 07dependabot[bot] commented on pull request 03#332: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/MirahezeMagic/pull/332#issuecomment-1082400799 [21:42:51] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 [21:42:53] [02MirahezeMagic] 07Universal-Omega deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/MirahezeMagic [21:43:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 8.78, 8.87 [21:44:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.64, 3.69 [21:46:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.84, 9.43, 9.60 [21:46:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.41, 8.32, 8.51 [21:48:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.87, 3.77 [21:48:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 8.82, 9.35 [21:49:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 8.90, 8.97 [21:50:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.60, 3.41 [21:50:33] miraheze/MirahezeMagic - dependabot[bot] the build passed. [21:51:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.55, 3.68 [21:51:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 8.65, 8.86 [21:52:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.10, 7.86, 8.36 [21:54:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.79, 3.58 [21:56:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 9.45, 9.39 [21:57:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.28, 9.02 [21:58:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.44, 3.73 [21:58:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 3.14, 3.36 [21:58:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.53, 8.98, 9.23 [21:59:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.48, 3.76 [21:59:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.92, 8.80, 8.88 [22:04:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.37, 10.62, 9.73 [22:05:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 2.90, 3.39 [22:06:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.17, 9.70, 9.51 [22:06:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.45, 9.38, 8.73 [22:08:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 2.91, 3.33 [22:08:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.63, 8.44, 8.45 [22:09:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 8.61, 8.60 [22:11:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.86, 8.38, 8.53 [22:13:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.46, 8.05, 8.39 [22:15:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.27, 9.95, 9.17 [22:17:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 9.66, 9.18 [22:19:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.11, 10.17, 9.08 [22:21:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.63, 9.43, 8.92 [22:24:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.60, 7.61, 8.44 [22:25:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.12, 9.74, 9.35 [22:25:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.67, 8.03, 8.47 [22:26:14] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ErrorPages/compare/660b41eb8074...5eba40a75116 [22:26:15] [02miraheze/ErrorPages] 07Universal-Omega 035eba40a - Create dependabot.yml [22:26:43] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/landing/compare/88b275c9e4bc...bf07550d9740 [22:26:45] [02miraheze/landing] 07Universal-Omega 03bf07550 - Create dependabot.yml [22:26:46] [02miraheze/ErrorPages] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/rectalogic/notify-irc-2 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/commit/6a9660ba0b3a [22:26:48] [02miraheze/ErrorPages] 07dependabot[bot] 036a9660b - Bump rectalogic/notify-irc from 1 to 2 [22:26:49] [02ErrorPages] 07dependabot[bot] created branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/ErrorPages [22:26:51] [02miraheze/ErrorPages] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-3 [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/commit/d77af30493ea [22:26:52] [02miraheze/ErrorPages] 07dependabot[bot] 03d77af30 - Bump actions/checkout from 2 to 3 [22:26:54] [02ErrorPages] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/ErrorPages [22:26:55] [02ErrorPages] 07dependabot[bot] opened pull request 03#6: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/ErrorPages/pull/6 [22:26:57] [02ErrorPages] 07dependabot[bot] opened pull request 03#5: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/ErrorPages/pull/5 [22:26:58] [02ErrorPages] 07dependabot[bot] labeled pull request 03#6: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/ErrorPages/pull/6 [22:27:00] [02ErrorPages] 07dependabot[bot] labeled pull request 03#5: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/ErrorPages/pull/5 [22:27:09] miraheze/ErrorPages - Universal-Omega the build passed. [22:27:19] [02miraheze/landing] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-3 [+0/-0/±1] 13https://github.com/miraheze/landing/commit/66fb81bb5854 [22:27:20] [02miraheze/landing] 07dependabot[bot] 0366fb81b - Bump actions/checkout from 2 to 3 [22:27:22] [02landing] 07dependabot[bot] opened pull request 03#75: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/landing/pull/75 [22:27:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.91, 9.63, 9.35 [22:27:23] [02landing] 07dependabot[bot] created branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/landing [22:27:25] [02landing] 07dependabot[bot] labeled pull request 03#75: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/landing/pull/75 [22:27:26] [02miraheze/landing] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/rectalogic/notify-irc-2 [+0/-0/±1] 13https://github.com/miraheze/landing/commit/e731a3962188 [22:27:28] [02miraheze/landing] 07dependabot[bot] 03e731a39 - Bump rectalogic/notify-irc from 1 to 2 [22:27:29] [02landing] 07dependabot[bot] created branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/landing [22:27:31] [02landing] 07dependabot[bot] opened pull request 03#76: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/landing/pull/76 [22:27:32] [02landing] 07dependabot[bot] labeled pull request 03#76: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/landing/pull/76 [22:27:39] miraheze/landing - Universal-Omega the build passed. [22:27:41] miraheze/ErrorPages - dependabot[bot] the build passed. [22:27:48] miraheze/ErrorPages - dependabot[bot] the build passed. [22:27:57] [02ErrorPages] 07Universal-Omega closed pull request 03#5: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/ErrorPages/pull/5 [22:27:59] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/5eba40a75116...81c851c794df [22:28:00] [02miraheze/ErrorPages] 07dependabot[bot] 0381c851c - Bump rectalogic/notify-irc from 1 to 2 (#5) [22:28:02] [02miraheze/ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 [22:28:03] [02ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/ErrorPages [22:28:13] miraheze/landing - dependabot[bot] the build passed. [22:28:18] [02landing] 07Universal-Omega closed pull request 03#75: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/landing/pull/75 [22:28:20] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/bf07550d9740...e024998224ff [22:28:21] [02miraheze/landing] 07dependabot[bot] 03e024998 - Bump actions/checkout from 2 to 3 (#75) [22:28:22] [02landing] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/landing [22:28:24] [02miraheze/landing] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/checkout-3 [22:28:29] miraheze/landing - dependabot[bot] the build passed. [22:28:40] [02landing] 07Universal-Omega closed pull request 03#76: Bump rectalogic/notify-irc from 1 to 2 - 13https://github.com/miraheze/landing/pull/76 [22:28:41] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/e024998224ff...90b23c920e4e [22:28:43] [02miraheze/landing] 07dependabot[bot] 0390b23c9 - Bump rectalogic/notify-irc from 1 to 2 (#76) [22:28:44] [02miraheze/landing] 07Universal-Omega deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 [22:28:46] [02landing] 07Universal-Omega deleted branch 03dependabot/github_actions/rectalogic/notify-irc-2 - 13https://github.com/miraheze/landing [22:28:53] miraheze/ErrorPages - Universal-Omega the build passed. [22:29:00] [02ErrorPages] 07Universal-Omega closed pull request 03#6: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/ErrorPages/pull/6 [22:29:02] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/81c851c794df...162b6c39c392 [22:29:03] [02miraheze/ErrorPages] 07dependabot[bot] 03162b6c3 - Bump actions/checkout from 2 to 3 (#6) [22:29:05] [02ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/checkout-3 - 13https://github.com/miraheze/ErrorPages [22:29:06] [02miraheze/ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/checkout-3 [22:29:15] miraheze/landing - Universal-Omega the build passed. [22:29:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.59, 8.30, 8.52 [22:29:31] miraheze/landing - Universal-Omega the build passed. [22:29:53] miraheze/ErrorPages - Universal-Omega the build passed. [22:31:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.17, 8.14, 8.43 [22:31:37] [02ssl] 07Universal-Omega opened pull request 03#495: Create dependabot.yml - 13https://github.com/miraheze/ssl/pull/495 [22:32:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [22:33:06] [02phabricator-extensions] 07Universal-Omega opened pull request 03#21: Create dependabot.yml - 13https://github.com/miraheze/phabricator-extensions/pull/21 [22:34:19] miraheze/phabricator-extensions - Universal-Omega the build passed. [22:35:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.37, 9.36, 8.79 [22:36:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:37:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 9.24, 8.82 [22:37:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.46, 9.21 [22:39:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 8.88, 8.65 [22:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:41:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.12, 8.75 [22:43:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.02, 9.08, 9.21 [22:43:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.74, 8.76, 8.65 [22:44:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 8.44, 8.06 [22:46:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.07, 8.37, 8.08 [22:46:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.39, 3.19 [22:48:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.98, 7.93, 8.48 [22:48:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.23, 3.16 [22:51:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.37, 8.04, 8.38 [22:54:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.38, 2.93 [22:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:54:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 4.38, 3.62 [22:54:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.64, 8.78, 8.36 [22:55:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.93, 6.88, 8.14 [22:56:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.84, 3.52 [22:58:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 2.93, 2.86 [22:58:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 3.22, 3.33 [22:58:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:14] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [23:02:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.87, 5.88, 3.10 [23:02:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.23, 8.05, 8.25 [23:04:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.50, 8.58, 8.26 [23:05:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.13, 2.90 [23:06:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.58, 3.42 [23:06:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.82, 9.23, 8.53 [23:07:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.82, 4.28, 3.39 [23:07:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 3.80, 3.18 [23:08:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.67, 3.47 [23:08:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 8.86, 8.49 [23:09:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.82, 3.27 [23:10:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.86, 4.06, 3.66 [23:10:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.87, 8.19, 8.29 [23:11:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.24, 7.96 [23:12:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.49, 3.50 [23:13:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.15, 8.40, 8.06 [23:13:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03rsyslog-remote-support [+3/-3/±6] 13https://github.com/miraheze/puppet/commit/1fa21682cfe7 [23:13:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.95, 3.37, 3.21 [23:13:56] [02miraheze/puppet] 07paladox 031fa2168 - base: Add support for rsyslog logging remotely in syslog module [23:13:57] [02puppet] 07paladox created branch 03rsyslog-remote-support - 13https://github.com/miraheze/puppet [23:14:06] [02puppet] 07paladox opened pull request 03#2458: base: Add support for rsyslog logging remotely in syslog module - 13https://github.com/miraheze/puppet/pull/2458 [23:14:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 8.68, 8.54 [23:14:42] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.44, 1.18, 1.95 [23:15:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.58, 3.54 [23:16:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.13, 3.37 [23:16:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.03, 8.29 [23:17:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.24, 2.75, 3.24 [23:18:42] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.20, 0.64, 1.55 [23:19:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 1.61, 0.83 [23:20:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.17, 7.96, 8.02 [23:20:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.42, 8.23, 8.46 [23:21:19] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.66, 0.95 [23:21:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 3.47, 3.38 [23:21:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 8.67, 8.21 [23:23:19] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.48, 0.98 [23:23:25] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.90, 1.98, 1.29 [23:25:07] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1506s [23:25:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:23] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.30, 1.70, 1.28 [23:27:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.60, 3.49 [23:27:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.12, 8.55 [23:29:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.83, 3.06, 3.31 [23:29:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.57, 9.09, 8.63 [23:30:49] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1847s [23:30:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.04, 9.10, 8.49 [23:33:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 8.24, 8.11 [23:33:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.31, 9.37, 8.81 [23:34:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.19, 8.74, 8.42 [23:34:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.27, 7.53, 7.87 [23:35:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.03, 7.94, 8.33 [23:37:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.43, 8.39, 8.38 [23:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:39:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 9.74, 9.20 [23:41:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 10.20, 9.08 [23:44:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.21, 8.68, 8.47 [23:44:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:46:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.06, 8.43, 8.41 [23:50:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 9.09, 8.67 [23:51:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.25, 9.18, 9.31 [23:51:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.78, 9.17, 8.73 [23:54:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.01, 9.10 [23:56:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.35, 9.34, 9.23 [23:57:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 8.63, 8.23 [23:57:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.55, 9.93, 9.19 [23:59:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 9.63, 9.17