[00:00:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.68, 8.00, 8.37 [00:01:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.20, 1.79, 1.37 [00:01:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 2.98, 2.52 [00:01:44] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.27, 5.86, 4.68 [00:02:58] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.50, 1.73, 1.40 [00:03:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.40, 3.09, 2.62 [00:03:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.36, 7.99, 8.41 [00:03:44] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.30, 5.48, 4.71 [00:03:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 8.00, 8.43 [00:03:49] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:55] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.46, 1.34 [00:06:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 8.16, 7.63 [00:08:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.44, 8.05, 7.67 [00:09:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 8.87, 8.61 [00:10:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.92, 3.01 [00:10:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.63, 3.02 [00:11:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 9.15, 8.63 [00:11:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.91, 9.60, 8.92 [00:13:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 9.11, 8.81 [00:13:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.81, 3.16 [00:14:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.03, 3.39, 3.05 [00:14:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.72, 3.14 [00:15:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.45, 8.97 [00:16:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.67, 3.19 [00:17:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.93, 8.15, 8.36 [00:17:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.21, 9.42, 9.02 [00:17:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.14, 3.04 [00:18:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.65, 3.23 [00:19:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.65, 9.29, 8.32 [00:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:21:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 9.66, 8.58 [00:21:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.14, 9.17, 8.08 [00:21:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 7.98, 8.48 [00:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 9.71, 8.71 [00:23:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.75, 3.33 [00:24:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.60, 3.22 [00:24:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.85, 8.97, 8.56 [00:25:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.03, 8.72, 8.21 [00:26:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.68, 3.88, 3.36 [00:26:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.61, 8.21, 8.33 [00:27:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.28, 8.34, 8.11 [00:29:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 4.01, 3.56 [00:29:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 8.37, 8.35 [00:30:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.29, 7.62, 8.32 [00:30:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 9.03, 8.63 [00:31:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.78, 3.53 [00:31:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.40, 7.46, 8.01 [00:31:49] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:32:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 8.43, 8.46 [00:33:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.44, 2.93, 3.25 [00:34:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.85, 3.37, 3.52 [00:38:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 2.98, 3.31 [00:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.80, 8.35, 7.97 [00:42:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.57, 9.26, 8.65 [00:42:27] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:45:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 8.42, 8.11 [00:46:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.60, 3.98 [00:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:48:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.84, 9.33, 8.54 [00:49:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 8.87, 8.10 [00:50:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.39, 8.38, 8.50 [00:50:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 8.93, 8.48 [00:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:52:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.89, 8.04, 8.20 [00:53:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.55, 8.22, 8.00 [00:54:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.24, 9.02, 8.73 [00:56:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.71, 3.34 [01:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:02:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.95, 7.88, 8.41 [01:05:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 8.55, 8.18 [01:07:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.52, 7.84, 7.97 [01:08:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.89, 9.47, 8.84 [01:08:27] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [01:09:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 8.07, 7.93 [01:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:11:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.41, 8.30, 8.04 [01:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 9.25, 8.89 [01:15:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 8.72, 7.95 [01:16:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.54, 9.43, 9.01 [01:17:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.52, 8.58, 7.99 [01:18:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 8.97, 8.89 [01:19:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.09, 8.95, 8.18 [01:19:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.35, 8.52, 7.96 [01:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:21:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.34, 8.73, 8.20 [01:21:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.33, 8.32, 7.93 [01:23:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.12, 8.49, 8.06 [01:24:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.21, 7.90, 8.02 [01:32:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.57, 8.95, 8.75 [01:34:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.39, 8.92, 8.75 [01:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:37:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.38, 9.72, 8.41 [01:38:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.92, 10.41, 9.25 [01:39:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.61, 9.26, 8.43 [01:41:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.40, 8.39, 8.22 [01:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 9.73, 9.82 [01:52:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:55:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.30, 8.58, 7.96 [01:57:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:57:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.11, 8.33, 7.93 [01:58:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.26, 8.84, 9.19 [02:00:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.87, 9.17 [02:02:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 8.86, 8.13 [02:04:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 8.91, 8.25 [02:07:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.59, 8.43, 7.93 [02:08:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.93, 7.82, 7.76 [02:09:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.05, 8.50, 7.93 [02:10:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.64, 9.60, 9.33 [02:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:11:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 9.52, 8.39 [02:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.29, 9.48, 9.32 [02:13:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.68, 8.84, 8.28 [02:13:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.27, 8.56 [02:15:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 8.19, 8.10 [02:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:15:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.83, 8.97, 8.55 [02:16:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 9.21, 8.69 [02:17:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.80, 8.92 [02:18:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.40, 9.32, 8.80 [02:18:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.29, 9.74, 9.45 [02:20:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 9.64, 9.43 [02:23:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.66, 8.91, 8.47 [02:23:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.74, 9.63, 9.29 [02:24:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.32, 9.10, 8.81 [02:25:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 8.73, 8.70 [02:28:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.23, 8.39, 8.46 [02:30:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.43, 9.47, 9.10 [02:31:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.48, 7.91, 8.40 [02:32:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 9.07, 9.00 [02:32:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.39, 8.54, 8.51 [02:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:33:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 7.43, 8.36 [02:34:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.14, 9.41, 9.12 [02:34:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.71, 8.45, 8.49 [02:36:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.13, 9.17, 9.06 [02:38:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.57, 9.20 [02:42:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 8.52, 8.17 [02:42:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.16, 9.77, 8.97 [02:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:43:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 8.98, 8.48 [02:44:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.73, 9.88, 9.12 [02:44:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.54, 9.76, 9.49 [02:44:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.26, 8.57, 8.14 [02:45:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 8.70, 8.42 [02:45:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.30, 9.29, 8.79 [02:46:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.93, 10.00, 9.23 [02:46:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.52, 8.23, 8.16 [02:46:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.61, 7.97, 7.98 [02:47:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.89, 8.90, 8.72 [02:50:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 8.96, 8.47 [02:51:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.32, 9.13, 8.65 [02:51:44] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.75, 9.21, 8.81 [02:53:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.70, 9.13, 8.82 [02:53:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.30, 8.78 [02:54:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.85, 8.21, 8.32 [02:57:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 9.72, 9.77 [02:57:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.20, 9.28, 8.93 [02:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:59:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.48, 8.44, 8.65 [02:59:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.14, 7.66, 8.28 [03:02:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.45, 8.74, 8.79 [03:03:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 8.82, 8.60 [03:03:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.30, 8.52, 8.32 [03:05:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 2.98, 2.43 [03:05:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.21, 2.43, 1.60 [03:06:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 8.99, 8.30 [03:07:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 2.76, 2.41 [03:07:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 9.08, 8.82 [03:07:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.14, 1.93, 1.51 [03:08:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 8.92, 8.35 [03:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:09:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.54, 1.51, 1.41 [03:09:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 9.37, 8.73 [03:10:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.23, 8.35, 8.21 [03:11:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 9.27, 8.89 [03:13:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.49, 9.04 [03:13:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 9.94, 9.17 [03:14:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.74, 9.40, 9.37 [03:15:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.45, 7.90, 8.40 [03:17:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.22, 10.02, 9.32 [03:19:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.13, 9.41, 9.19 [03:23:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.29, 7.72, 8.46 [03:25:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.26, 1.71, 1.08 [03:27:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.93, 1.35, 1.02 [03:29:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.30, 7.17, 8.21 [03:29:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.25, 7.56, 8.38 [03:32:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.19, 9.32, 8.48 [03:32:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.68, 8.69, 8.72 [03:34:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.09, 8.50, 8.29 [03:34:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 8.84, 8.78 [03:34:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.70, 9.29, 8.65 [03:36:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.96, 9.55, 9.04 [03:36:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 8.96, 8.62 [03:38:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.04, 8.84, 8.46 [03:38:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.14, 8.04, 8.30 [03:38:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.60, 8.94, 8.87 [03:40:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 8.33, 8.25 [03:42:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.76, 8.38, 8.36 [03:42:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 9.25, 8.80 [03:42:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.50, 10.13, 9.33 [03:42:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.83, 9.40, 8.64 [03:44:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.22, 9.72, 9.31 [03:44:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 9.53, 8.79 [03:48:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.12, 8.72 [03:49:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.74, 8.81, 8.44 [03:49:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.82, 9.23 [03:50:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.41, 9.09, 8.78 [03:51:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.43, 9.18, 9.06 [03:53:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.36, 8.42, 8.40 [03:54:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.04, 8.72, 8.46 [03:54:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.15, 7.47, 8.41 [03:54:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 8.21, 8.49 [03:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:56:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.92, 7.91, 8.18 [04:01:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.23, 9.09, 8.84 [04:01:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 8.50, 8.27 [04:03:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.63, 8.55, 8.66 [04:03:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.39, 8.02, 8.12 [04:04:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:06:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 8.47, 8.30 [04:09:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.89, 9.51, 8.67 [04:10:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.57, 7.80, 8.11 [04:13:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.89, 1.42 [04:13:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.90, 9.59, 8.95 [04:15:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.50, 1.68, 1.40 [04:15:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.10, 7.34, 8.15 [04:19:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.17, 8.91, 8.52 [04:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:21:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.63, 8.41, 8.38 [04:21:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.79, 9.45, 9.00 [04:23:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.41, 9.20, 8.96 [04:31:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 9.26, 8.97 [04:33:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.79, 8.56, 7.88 [04:33:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.59, 9.01, 8.91 [04:35:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.46, 8.34, 7.89 [04:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:35:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.06, 9.54, 9.13 [04:37:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.81, 8.96, 8.97 [04:39:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.96, 8.69, 8.13 [04:41:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.33, 9.37, 8.45 [04:41:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 9.05, 8.64 [04:43:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 9.35, 8.56 [04:45:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.25, 9.43, 8.68 [04:46:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.50, 9.02, 8.63 [04:47:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.50, 1.32 [04:47:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 8.18, 8.39 [04:49:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.41, 1.31 [04:50:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 8.68, 8.02 [04:52:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.07, 8.22 [04:52:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.45, 8.47, 8.50 [04:53:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.91, 7.92, 8.35 [04:54:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.71, 8.37, 8.05 [04:55:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.99, 8.09, 8.50 [05:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:11:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:16:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.63, 2.83 [05:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:18:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.55, 2.90 [05:19:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.36, 1.90, 1.38 [05:21:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.52, 1.46, 1.28 [05:21:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.36, 8.42, 7.89 [05:22:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 2.85, 2.78 [05:23:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:23:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 8.01, 7.82 [05:28:37] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:28:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.15, 8.96, 8.31 [05:30:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.06, 8.27, 8.13 [06:12:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.62, 8.74, 7.72 [06:14:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.44, 8.98, 7.93 [06:22:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.24, 9.03, 8.27 [06:26:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.93, 8.60, 8.35 [06:28:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.70, 8.34, 8.29 [06:34:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 8.07, 7.56 [06:35:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:37:20] PROBLEM - db111 APT on db111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [06:39:21] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 10 packages available for upgrade (1 critical updates). [06:42:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 8.26, 7.92 [06:45:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:47:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.44, 9.62, 8.68 [06:49:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.56, 8.11, 7.31 [06:51:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 8.64, 7.58 [06:53:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.25, 8.00, 8.31 [06:53:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.09, 8.40, 7.65 [06:53:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.89, 1.62, 1.13 [06:55:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.48, 1.53, 1.16 [07:09:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.75, 3.65, 2.55 [07:09:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.09, 2.39 [07:10:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.36, 8.78, 7.79 [07:11:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.20, 2.52 [07:11:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.08, 2.49 [07:12:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 8.51, 7.80 [07:14:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 7.75, 7.61 [07:24:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:36:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.19, 8.64, 7.79 [07:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:38:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.11, 8.17, 7.72 [07:42:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.38, 7.49 [07:43:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.21, 8.51, 7.80 [07:44:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.46, 9.22, 7.93 [07:45:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.60, 9.73, 8.33 [07:46:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 8.17, 7.19 [07:47:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.28, 9.21, 8.39 [07:47:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.95, 9.16, 8.30 [07:48:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.93, 8.47, 7.85 [07:48:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.97, 7.87, 7.19 [07:48:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.96, 9.36, 8.27 [07:53:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.23, 2.90 [07:53:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 8.76, 7.83 [07:55:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.18, 2.76 [07:55:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 2.84, 2.79 [07:56:47] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.49, 8.10, 8.21 [07:57:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.38, 3.01, 2.74 [07:57:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.16, 7.68, 7.63 [07:58:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.47, 8.27, 8.38 [07:59:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.30, 8.22, 8.33 [08:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:06:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 8.71, 8.49 [08:08:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.50, 8.01, 8.26 [08:12:27] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:17:56] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.54, 1.09 [08:18:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.16, 8.58, 8.41 [08:19:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.54, 8.56, 7.67 [08:21:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 8.61, 7.79 [08:21:52] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.78, 1.21, 1.06 [08:22:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.29, 8.11, 8.30 [08:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:23:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.89, 7.90, 7.63 [08:33:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:34:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.40, 2.01, 1.38 [08:36:24] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.62, 1.31 [08:40:27] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:41:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.84, 7.96, 7.72 [08:43:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.48, 7.63, 7.65 [09:52:05] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:54:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:01:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.89, 1.67, 1.43 [10:03:53] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.45, 1.57, 1.43 [10:09:41] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.62, 2.03, 1.64 [10:11:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.99, 1.67 [10:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.66, 7.87, 7.21 [10:13:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.25, 1.65, 1.58 [10:14:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.36, 7.57, 7.20 [10:20:05] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [10:21:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.55, 8.13, 7.55 [10:23:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.81, 7.37, 7.35 [11:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:10:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.19, 8.22, 7.20 [11:12:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.86, 7.67, 7.12 [11:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.94, 8.47, 7.33 [11:14:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 7.92, 7.27 [11:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:38:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.63, 1.32 [11:40:10] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.24, 1.40, 1.27 [11:53:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:57:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.76, 1.46 [11:59:06] [02miraheze/puppet] 07JohnFLewis pushed 036 commits to 03master [+16/-15/±25] 13https://github.com/miraheze/puppet/compare/b71a1d19265d...0dc92cd997f9 [11:59:07] [02miraheze/puppet] 07JohnFLewis 03371040a - prometheus: rewrite for prometheus101 [11:59:09] [02miraheze/puppet] 07JohnFLewis 03aff6f96 - fixes [11:59:10] [02miraheze/puppet] 07JohnFLewis 03d8e2e1e - more fixes for yaml [11:59:12] [02miraheze/puppet] ... and 3 more commits. [11:59:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.09, 1.51, 1.40 [12:03:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:05:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.31, 2.02, 1.66 [12:05:44] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.33, 6.69, 4.45 [12:07:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.39, 1.81, 1.62 [12:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.56, 1.55 [12:09:44] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.41, 6.30, 4.89 [12:13:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.25, 9.39, 8.02 [12:14:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 8.47, 7.85 [12:28:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.93, 10.50, 8.06 [12:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:34:55] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:12] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.011 second response time [12:35:39] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:46] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:58] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:56] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.143 second response time [12:39:27] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22220 bytes in 8.913 second response time [12:40:09] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.037 second response time [12:40:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.95, 8.43, 7.71 [12:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:43:40] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.06, 7.87, 7.67 [12:44:31] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:14] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:50:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:52:01] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [12:54:00] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22220 bytes in 0.426 second response time [12:54:01] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [12:54:13] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.025 second response time [12:54:30]