[00:04:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.85, 9.70, 9.06 [00:05:07] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:05:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.19, 10.20, 9.67 [00:08:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 9.41, 9.11 [00:08:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 8.67, 8.24 [00:09:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.09, 10.00, 9.25 [00:09:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.15, 10.23, 9.37 [00:11:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.27, 9.08, 8.47 [00:15:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 4.31, 3.36 [00:15:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.72, 9.91, 8.94 [00:18:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.63, 9.55, 9.17 [00:18:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.02, 10.07, 9.12 [00:19:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.82, 9.29, 9.71 [00:21:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.64, 3.44 [00:22:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.64, 9.73 [00:23:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.67, 1.85, 1.42 [00:23:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.75, 9.76, 9.67 [00:24:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.59, 3.43 [00:25:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.32, 9.75, 9.29 [00:26:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 9.33, 9.61 [00:26:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.64, 1.42 [00:26:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.68, 10.69, 10.05 [00:28:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.47, 3.40 [00:28:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.15, 9.96, 9.77 [00:29:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 9.78, 9.73 [00:29:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.49, 9.84, 9.83 [00:31:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.35, 2.80, 3.15 [00:31:34] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:32:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.71, 10.25, 9.95 [00:32:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 9.72, 9.73 [00:32:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.42, 10.26, 9.98 [00:35:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 9.96, 9.91 [00:35:37] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.74, 9.71 [00:40:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 9.40, 9.77 [00:43:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.18, 9.61, 9.77 [00:44:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.63, 9.25 [00:46:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 9.22, 9.61 [00:48:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 9.31, 9.19 [00:51:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.39, 9.64, 9.52 [00:51:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.06, 9.64, 9.93 [00:53:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.28, 10.06, 9.83 [00:54:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.90, 9.35, 9.45 [00:54:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.63, 9.94, 9.97 [00:56:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.76, 9.24, 9.57 [00:58:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.30, 9.88, 9.65 [00:58:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.97, 9.83, 9.92 [00:59:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.91, 10.01, 9.81 [01:01:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 9.86, 9.71 [01:02:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.95, 9.00, 9.47 [01:05:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.86, 10.06, 9.80 [01:07:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.98, 10.19, 9.47 [01:08:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 9.93, 9.80 [01:10:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.75, 9.67 [01:14:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.56, 2.03, 1.69 [01:14:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 9.80, 9.73 [01:14:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.90, 9.40, 9.78 [01:15:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.18, 10.46, 10.04 [01:17:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.58, 1.58 [01:17:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.98, 9.85, 9.59 [01:17:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.06, 9.36, 9.29 [01:18:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.35, 9.29, 9.64 [01:20:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.52, 10.46, 9.93 [01:20:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.99, 9.80, 9.69 [01:21:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.83, 10.30, 10.09 [01:21:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.64, 9.69 [01:24:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.96, 9.47, 9.36 [01:24:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 2.13, 1.57 [01:24:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 9.18, 9.84 [01:24:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.98, 9.10, 9.66 [01:25:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 4.60, 3.57 [01:25:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 4.19, 3.41 [01:27:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.58, 9.37, 9.66 [01:27:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.59, 1.38, 1.37 [01:27:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 4.84, 3.45 [01:27:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 9.73, 9.78 [01:30:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.08, 9.76, 9.76 [01:31:05] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.33, 10.00, 9.89 [01:33:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.10, 9.69, 9.75 [01:34:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 10.48, 9.95 [01:35:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.51, 3.61 [01:35:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.71, 3.70 [01:37:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.03, 9.64, 9.71 [01:37:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.88, 3.36, 3.37 [01:37:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 9.53, 9.81 [01:40:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.14, 9.63, 9.37 [01:40:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.22, 9.31, 9.60 [01:40:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.29, 9.38 [01:41:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.63, 9.78 [01:42:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 3.05, 3.38 [01:43:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 10.09, 9.85 [01:43:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 8.75, 9.12 [01:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 8.85, 9.43 [01:45:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.21, 3.38 [01:46:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 9.98, 9.71 [01:46:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.79, 9.44, 9.64 [01:48:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.07, 3.29 [01:50:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.79, 3.69 [01:50:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.72, 10.54, 9.75 [01:50:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.46, 3.32 [01:50:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.89, 11.09, 10.12 [01:54:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.83, 3.10, 3.20 [01:56:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.47, 3.52 [01:56:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.27, 9.60, 9.43 [01:57:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 9.63, 9.75 [01:57:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.86, 8.79, 9.56 [01:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:59:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.15, 3.06, 3.34 [01:59:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.63, 9.44 [01:59:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.87, 9.21, 9.31 [02:00:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.16, 8.89, 9.53 [02:00:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.78, 9.18, 9.81 [02:03:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.13, 9.46, 9.46 [02:03:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 9.60, 10.07, 9.67 [02:03:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.55, 10.45, 10.13 [02:03:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:03:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.58, 9.37, 9.43 [02:06:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.46, 9.34, 9.42 [02:06:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.99, 9.35, 9.39 [02:09:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.96, 9.93, 9.51 [02:09:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.63, 9.02, 9.66 [02:12:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.72, 3.49 [02:12:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 9.99, 9.56 [02:13:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.10, 10.03, 9.95 [02:13:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.40, 8.85, 8.93 [02:16:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.42, 10.43, 9.75 [02:18:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.77, 2.60, 3.10 [02:19:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 9.77, 9.63 [02:19:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.10, 9.76, 9.63 [02:22:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.09, 9.20, 9.73 [02:22:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.35, 8.87, 9.01 [02:25:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.74, 9.78, 9.85 [02:25:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.30, 9.04, 9.18 [02:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.60, 8.63, 9.42 [02:29:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.56, 9.28, 9.30 [02:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:32:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.10, 9.43 [02:33:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.51, 3.37 [02:34:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.23, 8.72, 9.59 [02:34:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.11, 10.10, 9.81 [02:36:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.87, 3.54 [02:38:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.69, 3.29 [02:38:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.40, 9.07, 9.38 [02:38:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.14, 7.93, 8.41 [02:39:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.70, 3.55 [02:41:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.34, 9.78, 9.67 [02:41:33] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.78, 2.72, 2.98 [02:42:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.94, 3.69 [02:44:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.38, 9.70, 9.50 [02:45:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.51, 3.56 [02:48:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.54, 9.22, 9.37 [02:48:05] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.79, 8.03, 8.37 [02:53:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.52, 9.65, 9.98 [02:53:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.74, 8.98, 9.57 [02:55:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 2.75, 3.25 [02:55:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.12, 8.32, 8.51 [02:56:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.10, 10.19, 10.12 [02:57:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.77, 7.58, 8.46 [02:59:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.28, 7.89, 8.32 [02:59:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.77, 9.26, 8.87 [03:00:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.97, 7.90, 8.39 [03:02:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.50, 9.45, 8.98 [03:05:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.52, 8.46, 8.73 [03:05:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.18, 8.20, 8.58 [03:05:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.08, 8.88, 9.65 [03:08:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.00, 8.87 [03:08:35] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.14, 7.63, 8.28 [03:11:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.47, 8.97, 8.67 [03:11:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.24, 8.19, 8.59 [03:12:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 7.61, 8.30 [03:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:17:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.99, 7.99, 8.39 [03:17:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.73, 7.75, 8.24 [03:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 9.00, 8.96 [03:30:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.31, 8.92, 9.03 [03:38:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 9.64, 8.63 [03:40:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.38, 10.29, 9.69 [03:40:49] [02mw-config] 07lens0021 synchronize pull request 03#4473: Add wgPageForms{RenameEditTabs,RenameMainEditTab,LinkAllRedLinksToFor… - 13https://github.com/miraheze/mw-config/pull/4473 [03:41:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.16, 9.37, 8.74 [03:41:50] miraheze/mw-config - lens0021 the build passed. [03:41:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.13, 9.76, 8.20 [03:42:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.75, 9.28, 8.45 [03:43:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.58, 9.57, 9.53 [03:45:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.40, 8.60, 8.06 [03:46:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.63, 9.31, 9.39 [03:48:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.22, 8.38 [03:51:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.49, 8.19, 8.48 [03:52:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.63, 9.79, 9.71 [03:54:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.95, 9.92, 9.17 [03:54:51] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 7.91, 8.27 [03:55:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.86, 10.28, 9.88 [03:57:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 10.25, 9.47 [03:59:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 9.41, 8.90 [03:59:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.19, 8.75, 8.29 [04:01:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.13, 9.50, 9.36 [04:02:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.21, 9.93, 9.22 [04:02:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.24, 7.90, 8.07 [04:04:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.40, 9.63, 9.41 [04:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:05:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.05, 8.91, 8.54 [04:07:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 9.95, 9.58 [04:08:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.38, 9.92, 9.79 [04:08:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 9.44, 9.25 [04:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:12:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.27, 8.10, 8.27 [04:15:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.04, 9.32, 8.72 [04:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.22, 7.76, 8.49 [04:27:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.36, 7.82, 8.43 [04:27:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 7.61, 8.29 [04:33:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.17, 9.14 [04:35:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.19, 8.55, 8.54 [04:35:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 9.61, 9.00 [04:37:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 8.85, 8.62 [04:38:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.37, 9.70, 8.97 [04:39:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.72, 9.93, 9.53 [04:40:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.65, 9.03, 8.33 [04:43:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.05, 8.58, 8.34 [04:44:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.81, 7.91, 8.34 [04:44:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.49, 8.71, 8.83 [04:45:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.90, 2.14, 1.55 [04:46:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.14, 7.43, 7.92 [04:51:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.80, 7.96, 8.42 [04:51:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.81, 7.46, 8.32 [04:55:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.66, 7.58, 8.42 [04:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:02:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:02:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.45, 8.64, 8.70 [05:06:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.31, 10.35, 9.40 [05:09:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.93, 9.80, 9.34 [05:12:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.13, 9.87, 8.62 [05:12:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.82, 9.42 [05:13:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 9.36, 8.57 [05:15:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.70, 8.99, 8.53 [05:15:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 9.56, 9.39 [05:19:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.74, 8.35, 8.36 [05:25:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.22, 8.04, 8.39 [05:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:28:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.04, 9.65, 9.36 [05:31:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 9.55, 9.38 [05:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:37:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 9.56, 8.79 [05:38:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.19, 8.55, 8.54 [05:40:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.79, 9.30, 8.84 [05:43:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.38, 7.78, 8.48 [05:44:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.60, 7.76, 8.28 [05:45:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 8.67, 8.34 [05:48:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 8.03, 8.15 [05:53:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.67, 8.09, 8.39 [06:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:17:47] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 7.724 second response time [06:18:43] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.805 second response time [06:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:25:49] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:52] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 9.19, 8.03 [06:38:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.20, 7.89, 8.03 [06:38:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.75, 9.36, 8.69 [06:50:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.76, 8.91, 8.17 [06:52:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.24, 9.18, 8.77 [06:52:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.03, 9.05, 8.35 [06:56:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.55, 8.20, 8.14 [06:58:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.95, 8.02, 8.18 [07:03:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 8.60, 8.22 [07:05:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.65, 7.87, 8.33 [07:06:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.50, 7.61, 7.94 [07:12:43] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/py [07:12:43] st-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query fbpml.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4 [07:12:43] ::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:15:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.67, 9.91, 8.87 [07:15:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.77, 9.06, 8.61 [07:16:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.93, 8.82, 8.46 [07:18:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.73, 8.99, 8.71 [07:23:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 9.06, 8.55 [07:24:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.05, 7.91, 8.45 [07:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:28:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.78, 8.18, 8.47 [07:28:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.78, 8.44, 8.46 [07:33:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 8.11, 8.33 [07:39:06] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.388 second response time [07:39:35] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 6.480 second response time [07:40:42] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.557 second response time [07:41:37] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 7.868 second response time [07:42:56] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.24, 1.49, 1.87 [07:42:59] RECOVERY - wiki.fbpml.org - reverse DNS on sslhost is OK: SSL OK - wiki.fbpml.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:47:04] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:41] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:48:46] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:49:11] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.03, 1.17, 1.59 [07:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:49:46] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:59] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.673 second response time [07:59:53] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:25:23] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 3.660 second response time [08:25:42] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 2.944 second response time [08:25:57] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.683 second response time [08:26:44] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.787 second response time [08:27:00] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.295 second response time [08:27:38] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.26, 3.20, 2.10 [08:33:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:44:39] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:13] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:28] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:36] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:16] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.237 second response time [08:48:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:53:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:57:42] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 7.506 second response time [08:59:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:02:47] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 8.147 second response time [09:02:47] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 8.469 second response time [09:02:48] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 7.943 second response time [09:03:31] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.138 second response time [09:04:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:11:26] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:48] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:40] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:48] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:24] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:27] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.636 second response time [09:22:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:01] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.506 second response time [09:23:34] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 8.227 second response time [09:23:41] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 8.483 second response time [09:26:22] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 3.162 second response time [09:27:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:35:35] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:22] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:32] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:51] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:13] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:10] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 8.641 second response time [09:41:27] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 3.787 second response time [09:42:14] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.582 second response time [09:49:12] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:49:37] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:24] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:46] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.973 second response time [09:50:49] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 3.707 second response time [09:52:16] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.569 second response time [09:52:39] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 3.646 second response time [09:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:56:51] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 6.717 second response time [10:23:40] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:23] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:41] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:59] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:26:43] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.467 second response time [10:27:53] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.06, 1.29, 1.92 [10:33:57] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 2.933 second response time [10:34:00] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.00, 1.09, 1.61 [10:34:04] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 2.928 second response time [10:34:17] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.486 second response time [10:40:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.13, 7.40 [10:41:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.13, 8.66, 7.30 [10:43:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.51, 8.26, 7.44 [10:44:08] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:44:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.87, 7.82, 7.24 [10:45:20] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:27] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 6.772 second response time [10:54:19] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:22] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:42] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:12] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 9.382 second response time [10:55:28] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:57:22] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.218 second response time [10:57:24] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.595 second response time [10:57:45] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 1.916 second response time [10:58:31] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.099 second response time [11:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:37:08] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:37] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:46] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:59] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:49] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:43] [02mw-config] 07lens0021 opened pull request 03#4474: Add RelatedActicles configurations to ManageWikiSettings.php - 13https://github.com/miraheze/mw-config/pull/4474 [11:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:47:50] miraheze/mw-config - lens0021 the build passed. [11:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:01:57] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.001 second response time [12:02:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 9.13, 8.15 [12:03:05] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.788 second response time [12:05:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.10, 8.05, 7.90 [12:08:42] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/ee1e0fc319cf...3e173f6cb4fd [12:08:44] [02miraheze/IncidentReporting] 07translatewiki 033e173f6 - Localisation updates from https://translatewiki.net. [12:08:45] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e9da7c3e206c...2cfd74e04888 [12:08:47] [02miraheze/CreateWiki] 07translatewiki 032cfd74e - Localisation updates from https://translatewiki.net. [12:08:48]