[00:00:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.35, 3.03 [00:00:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.08, 8.25, 7.89 [00:03:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 690 failures. Last run 2 minutes ago with 690 failures. Failed resources (up to 3 shown): File[hi.gyaanipedia.com_private],File[pa.gyaanipedia.com],File[pa.gyaanipedia.com_private],File[mr.gyaanipedia.com] [00:03:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.93, 2.17, 1.64 [00:03:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 8.54, 7.95 [00:03:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.84, 4.49, 3.37 [00:03:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 8.74, 8.26 [00:04:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 3.89, 3.30 [00:04:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.03, 8.28, 7.99 [00:04:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.21, 4.60, 3.55 [00:05:29] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:05:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.19, 8.27, 7.94 [00:05:52] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.11, 8.44 [00:07:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.66, 1.84, 1.63 [00:07:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:09:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.97, 1.71 [00:10:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.57, 3.85, 3.56 [00:10:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 8.92, 8.28 [00:11:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.84, 1.70 [00:12:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.01, 8.47, 8.18 [00:12:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.75, 3.63 [00:13:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 2.12, 1.83 [00:13:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 4.00, 3.90 [00:14:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 4.20, 3.81 [00:15:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 4.28, 4.03 [00:15:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 4.03, 3.65 [00:17:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.67, 1.72 [00:17:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.59, 9.64, 9.32 [00:17:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.70, 3.86 [00:17:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.64, 3.53 [00:19:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.33, 1.59 [00:19:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 4.09, 3.98 [00:19:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.83, 3.92, 3.64 [00:21:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.74, 8.76 [00:21:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.99, 10.13, 9.52 [00:21:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.38, 3.49, 3.78 [00:21:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.31, 3.44 [00:22:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 9.28, 8.71 [00:23:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.68, 9.78, 9.49 [00:23:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.00, 3.31 [00:24:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.80, 3.93 [00:27:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.11, 9.68, 9.46 [00:28:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:29:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.87, 8.66, 9.12 [00:31:22] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:32:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.68, 3.74 [00:33:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.91, 8.28, 8.44 [00:33:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 9.87, 9.45 [00:33:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.98, 3.75 [00:34:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.35, 3.61 [00:35:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.98, 3.79 [00:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.81, 9.08, 9.29 [00:37:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.99, 4.09, 3.86 [00:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:38:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.68, 8.28, 7.84 [00:40:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.97, 7.94, 7.79 [00:40:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 3.71, 3.61 [00:42:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.43, 3.51 [00:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.72, 4.16, 3.76 [00:45:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.14, 7.63, 8.46 [00:46:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.74, 3.65 [00:49:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.56, 9.07 [00:50:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.37, 2.03, 1.65 [00:51:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.95, 8.57, 8.77 [00:53:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:53:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.88, 8.50, 7.75 [00:54:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.10, 3.37 [00:55:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.81, 7.84, 8.46 [00:57:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.88, 9.46, 8.26 [00:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.39, 8.81, 8.71 [00:59:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.44, 9.29, 8.35 [01:00:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.45, 3.41 [01:01:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 8.93, 8.42 [01:03:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.55, 8.48, 8.32 [01:05:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.18, 7.93, 8.11 [01:07:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 8.96, 8.73 [01:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:08:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 3.07, 3.29 [01:09:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.82, 8.51, 8.60 [01:09:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 9.33, 8.67 [01:11:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.61, 8.09, 8.42 [01:11:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.80, 8.54 [01:11:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.45, 3.93 [01:11:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 9.09, 8.59 [01:13:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.23, 3.06 [01:14:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.75, 1.97 [01:14:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 4.09, 3.67 [01:15:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.16, 3.06 [01:16:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.79, 3.60 [01:19:37] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.64, 9.21, 8.80 [01:19:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.67, 8.33, 8.47 [01:19:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.97, 2.00 [01:21:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.78, 9.28, 8.87 [01:22:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 8.76, 8.60 [01:23:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.81, 2.65, 3.28 [01:25:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:25:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.46, 8.08, 8.48 [01:26:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 7.75, 7.33 [01:26:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.25, 4.22, 3.73 [01:28:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.31, 7.86, 7.42 [01:29:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.69, 8.45, 8.59 [01:29:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.91, 4.11 [01:30:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.85, 4.54, 3.74 [01:31:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.25, 8.12, 8.46 [01:31:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.79, 1.96 [01:32:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.58, 3.83, 3.57 [01:33:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.68, 3.77 [01:34:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.30, 8.95 [01:35:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:35:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 4.00, 3.87 [01:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.04, 8.54, 8.72 [01:37:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.52, 3.70 [01:39:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.44, 2.17, 2.04 [01:40:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 2.99, 3.30 [01:40:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.43, 3.78 [01:42:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.45, 7.45, 8.25 [01:43:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 2.92, 3.36 [01:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.79, 3.82 [01:47:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.68, 3.27, 3.46 [01:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.47, 3.72 [01:49:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.55, 3.04, 3.35 [01:56:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.07, 3.38 [01:57:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.57, 3.46 [01:59:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.12, 3.16, 3.33 [02:00:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.52, 8.41, 8.04 [02:04:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.13, 8.40, 8.15 [02:07:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.62, 3.51 [02:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:09:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.67, 5.53, 4.23 [02:10:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.35, 4.16, 3.53 [02:14:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.91, 3.54 [02:15:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.95, 3.67, 3.86 [02:17:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.91, 3.92 [02:22:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.21, 3.29, 3.38 [02:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:34:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 7.99, 7.66 [02:36:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.35, 7.42, 7.50 [02:38:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.82, 3.52 [02:40:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.54, 3.30 [02:42:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.32, 3.25 [02:42:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.13, 8.27, 7.78 [02:42:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.22, 3.34 [02:42:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 8.55, 7.38 [02:43:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.16, 1.99 [02:44:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.45, 8.26, 7.83 [02:46:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.53, 7.47, 7.26 [02:47:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.56, 0.82, 1.66 [02:52:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.50, 3.40 [02:55:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.01, 8.12, 7.68 [02:55:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.20, 3.26, 3.31 [02:56:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.61, 9.19, 8.38 [02:57:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.63, 8.37, 7.76 [02:57:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.45, 8.21, 7.76 [02:57:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.42, 9.06, 8.45 [02:59:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.78, 8.12, 7.75 [02:59:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.72, 8.77 [03:01:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.89, 9.06, 8.65 [03:03:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.82, 8.39, 8.46 [03:08:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.21, 9.15, 8.46 [03:10:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.72, 8.08, 8.17 [03:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:14:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 4.18, 3.56 [03:16:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.98, 3.57 [03:18:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.72, 4.44, 3.77 [03:20:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.59, 3.13 [03:23:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 3.12, 3.02 [03:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.59, 3.71 [03:28:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.80, 3.76 [03:29:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 8.45, 8.16 [03:30:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.83, 3.78 [03:31:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.41, 7.72, 7.93 [03:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:35:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.74, 8.60, 8.25 [03:37:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.31, 7.75, 7.98 [03:38:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 2.91, 3.38 [03:43:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:46:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.88, 3.57 [03:48:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.52, 3.48 [03:49:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.27, 9.06, 8.42 [03:50:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 3.03, 3.30 [03:51:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.77, 8.68, 8.25 [03:51:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 8.95, 8.45 [03:51:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.38, 3.24, 3.88 [03:52:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:52:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.15, 7.93, 8.01 [03:53:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.25, 8.44, 8.34 [03:58:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.88, 3.43, 3.30 [04:00:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.10, 3.20 [04:02:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:03:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 2.77, 3.34 [04:08:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:12:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.84, 1.38, 1.11 [04:14:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.82, 1.22, 1.09 [04:18:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.08, 8.87, 8.11 [04:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:20:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.94, 8.30, 7.98 [04:23:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.81, 8.54, 8.15 [04:26:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.04, 8.01, 8.00 [04:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:31:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.91, 3.40 [04:33:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.75, 3.18, 3.20 [04:35:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:37:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 8.12, 7.33 [04:39:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.84, 7.95, 7.36 [04:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:46:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.74, 3.21 [04:48:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.48, 3.17 [04:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:50:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.67, 3.32, 3.16 [04:52:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 8.90, 7.89 [04:55:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.18, 9.10, 8.12 [04:57:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.16, 8.70, 8.09 [04:58:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.37, 8.79, 8.33 [04:59:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.22, 8.07, 7.92 [05:04:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.43, 8.23, 8.26 [05:05:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 8.68, 8.28 [05:07:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.66, 8.21, 8.15 [05:08:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.14, 3.03 [05:09:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 3.86, 3.30 [05:10:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.38, 3.11, 3.03 [05:11:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.74, 3.32 [05:13:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.29, 3.20 [05:14:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 8.43, 7.56 [05:15:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.26, 8.55, 8.24 [05:16:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 7.99, 7.52 [05:17:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:17:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.13, 8.09, 8.12 [05:22:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:34:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:38:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.06, 1.29, 1.99 [05:39:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:42:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.13, 2.77 [05:44:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.11, 2.97, 2.74 [05:46:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.18, 1.19, 1.65 [05:53:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.34, 2.90 [05:54:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.56, 3.03 [05:55:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 4.15, 3.28 [05:56:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 3.28, 2.99 [05:57:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.75, 3.24 [05:59:31] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.34, 2.10, 1.78 [05:59:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.86, 3.00, 3.02 [06:08:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:11:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.74, 8.92, 7.68 [06:12:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.87, 8.85, 7.83 [06:13:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:13:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.99, 8.03, 7.50 [06:14:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.02, 8.30, 7.73 [06:20:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.38, 8.30 [06:22:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 9.13, 8.34 [06:24:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 8.23, 8.10 [06:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:46:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:51:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:55:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.02, 8.54, 7.86 [06:57:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 8.45, 7.89 [06:59:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.71, 8.30, 7.92 [06:59:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:16:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:22:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.53, 2.68, 1.59 [07:22:37] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 34.69, 15.19, 8.17 [07:23:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:24:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.66, 1.97, 1.46 [07:24:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.43, 3.26 [07:26:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.18, 1.65, 1.39 [07:26:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.92, 3.09 [07:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:29:27] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:32:37] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.41, 7.45, 7.75 [07:36:37] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.28, 5.15, 6.79 [07:42:37] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.73, 6.83, 7.02 [07:44:37] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.76, 6.10, 6.73 [07:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:51:29] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.11, 6.78, 6.83 [07:53:24] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 21.37, 11.93, 8.64 [07:58:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.24, 1.65, 1.93 [07:59:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:06:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.48, 1.43, 1.68 [08:06:49] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.35, 5.80, 7.56 [08:08:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.83, 1.84, 1.31 [08:09:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:10:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.92, 1.51, 1.25 [08:10:39] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.08, 4.89, 6.80 [08:21:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.21, 2.63 [08:23:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.45, 3.16 [08:23:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.74, 2.53 [08:25:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.68, 2.86, 2.98 [08:33:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.91, 4.25, 3.45 [08:34:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:34:37] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 12.10, 7.75, 6.22 [08:35:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.81, 3.38 [08:37:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 3.25, 3.24 [08:38:37] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.67, 6.52, 6.11 [08:39:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:41:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.14, 3.42, 3.38 [08:44:37] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.41, 9.88, 7.95 [08:45:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.87, 3.18 [08:58:37] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.91, 7.22, 7.94 [09:00:37] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 10.72, 8.64, 8.36 [09:05:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:08:37] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.87, 7.05, 7.84 [09:10:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:14:37] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.20, 7.87, 7.86 [09:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:29:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.24, 2.84 [09:29:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:31:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.06, 2.85, 2.75 [09:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:37:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.44, 3.69, 3.04 [09:39:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.55, 3.06 [09:41:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.01, 3.33, 3.04 [09:45:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.47, 3.18 [09:47:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 2.88, 3.00 [09:52:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.53, 8.02, 7.29 [09:52:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.34, 1.75, 1.41 [09:53:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:54:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.50, 1.68, 1.43 [09:56:00] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.65, 1.87, 1.19 [09:56:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.73, 8.01, 7.46 [09:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:59:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.54, 5.52, 3.93 [09:59:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 3.89, 2.93 [09:59:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.46, 2.66 [10:00:00] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.20, 1.93, 1.42 [10:01:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:02:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.69, 1.51, 1.33 [10:03:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.96, 3.09 [10:04:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.46, 2.64, 1.64 [10:05:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 4.23, 3.29 [10:05:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 8.27, 7.23 [10:06:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.03, 2.00, 1.53 [10:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:07:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.78, 3.24 [10:07:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.94, 7.73, 7.16 [10:08:02] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.cdntennis.ca All nameservers failed to answer the query. [10:08:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.55, 1.50, 1.40 [10:09:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.91, 3.55 [10:09:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.83, 3.32 [10:11:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.66, 3.30 [10:15:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 4.11, 3.72 [10:19:33] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:21:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 3.17, 3.30 [10:21:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.12, 3.93 [10:24:33] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:26:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.62, 3.82 [10:26:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.57, 3.46 [10:28:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.30, 3.38 [10:30:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 2.73, 3.40 [10:31:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.74, 2.52, 3.34 [10:32:39] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [10:32:40] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:36:34] RECOVERY - wiki.cdntennis.ca - reverse DNS on sslhost is OK: SSL OK - wiki.cdntennis.ca reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:36:37] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 1.50, 4.17, 7.27 [10:38:37] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.69, 3.27, 6.56 [10:46:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.00, 2.97 [10:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:48:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 2.75, 2.88 [10:50:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.44, 4.32, 3.64 [10:52:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 4.00, 3.60 [10:53:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:56:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.97, 4.13, 3.66 [10:59:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:02:13] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:04:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.77, 3.81 [11:05:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.73, 8.31, 7.45 [11:06:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.73, 3.78 [11:07:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.81, 8.10, 7.47 [11:08:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.26, 3.60 [11:09:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:10:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.90, 8.18, 7.62 [11:13:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 8.64, 7.76 [11:14:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.17, 7.29, 7.44 [11:15:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.73, 7.73, 7.51 [11:20:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.55, 2.74, 3.32 [11:21:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 8.57, 8.00 [11:25:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.63, 8.24, 8.00 [11:28:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.56, 3.24 [11:28:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 3.64, 3.41 [11:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:30:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.80, 3.36 [11:30:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.52, 3.38 [11:32:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.30, 3.22 [11:32:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.84, 4.49, 3.75 [11:36:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.67, 8.55, 8.09 [11:36:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.57, 3.31 [11:38:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.41, 3.28 [11:40:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.29, 8.35, 8.15 [11:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:40:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.04, 3.16 [11:40:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.90, 3.88 [11:48:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 2.96, 3.34 [11:58:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 7.82, 7.40 [11:58:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.53, 3.49 [12:00:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.25, 7.32, 7.28 [12:01:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:04:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.17, 3.38 [12:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:08:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.32, 3.41 [12:08:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.15, 1.71, 1.43 [12:10:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 3.68, 3.51 [12:12:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.19, 2.81 [12:12:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.98, 1.91, 1.59 [12:13:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.51, 3.14 [12:14:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.22, 2.02, 1.67 [12:16:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.42, 2.99 [12:18:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.15, 2.95 [12:18:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.74, 3.70 [12:19:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.11, 3.12 [12:20:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.28, 4.21, 3.87 [12:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:22:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.74, 3.40 [12:24:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.78, 3.76 [12:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:26:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.79, 4.79, 4.15 [12:28:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 3.35, 3.36 [12:31:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:32:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.91, 3.58 [12:35:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.56, 1.95, 1.31 [12:36:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.73, 3.31, 3.46 [12:37:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.62, 1.26 [12:38:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.45, 3.94 [12:38:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.88, 3.13, 3.37 [12:38:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.62, 8.48, 7.38 [12:40:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.36, 8.28, 7.43 [12:42:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.03, 4.38, 4.18 [12:42:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.44, 3.49 [12:44:35] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/nginx] [12:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 4.11, 3.73 [12:45:00] That's weird [12:46:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.92, 3.71 [12:47:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.88, 1.48 [12:48:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.83, 4.13, 3.83 [12:49:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.54, 1.40 [12:49:28] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [12:50:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.84, 3.77 [12:52:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.68, 4.00 [12:54:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.53, 4.39, 3.97 [12:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:05:26] !log tightened the high worker alert so it goes off less. Now alerts if over 90% for more than 7m checking every 90s [13:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:06:18] FYI Reception123 [13:08:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.13, 8.87, 8.18 [13:08:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.83, 3.97 [13:10:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 2.68, 3.32 [13:10:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 8.81, 8.23 [13:12:33] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:16:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.61, 3.71 [13:18:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.48, 8.82, 8.15 [13:20:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.92, 8.25, 8.01 [13:22:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.31, 8.20, 8.40 [13:27:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 3.81, 3.39 [13:29:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.51, 3.32 [13:33:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.58, 4.14, 3.58 [13:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:43:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.54, 3.67, 3.65 [13:43:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:44:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 8.47, 7.94 [13:45:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 4.29, 3.89 [13:46:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 8.24, 7.92 [13:46:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.17, 1.49, 1.95 [13:47:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.27, 3.12 [13:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:51:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.91, 3.91 [13:51:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.80, 3.38 [13:52:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.74, 8.43, 8.02 [13:53:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 4.38, 4.08 [13:54:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 8.32, 7.84 [13:55:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.07, 3.92, 3.58 [13:56:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.23, 8.87, 8.09 [13:56:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.19, 1.36, 1.66 [13:57:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.90, 3.94 [13:57:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 9.16, 8.51 [13:58:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.97, 8.44, 8.20 [13:58:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.80, 8.91, 8.21 [13:59:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.91, 4.66, 4.19 [13:59:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.70, 4.45, 3.79 [14:03:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 4.00, 3.76 [14:03:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.55, 9.41, 8.75 [14:03:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.41, 8.24, 8.16 [14:05:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.74, 9.02, 8.70 [14:07:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.94, 3.76 [14:10:59] [discord] CosmicAlpha, once you have enough time, Check DM [14:11:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.87, 3.80 [14:11:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.33, 7.65, 8.24 [14:13:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:15:12] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.68, 3.71 [14:17:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.77, 3.74 [14:20:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:20:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.55, 8.05 [14:22:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.74, 8.05, 7.92 [14:22:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 7.82, 7.85 [14:24:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 7.65, 7.78 [14:25:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:25:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.94, 3.81 [14:26:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.20, 3.27, 3.58 [14:28:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.74, 8.65, 8.08 [14:30:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.18, 8.20, 7.99 [14:32:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.00, 2.96, 3.40 [14:33:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.55, 8.43, 7.91 [14:34:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 8.32, 7.87 [14:35:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.44, 8.24, 7.91 [14:36:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.68, 7.89, 7.78 [14:36:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.88, 2.82, 3.88 [14:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:39:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.39, 8.43 [14:43:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.30, 8.80, 8.41 [14:44:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.40, 3.43, 3.95 [14:45:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.16, 8.45, 8.31 [14:48:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:48:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 3.88, 3.76 [14:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 8.72, 8.04 [14:52:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.80, 4.37, 4.13 [14:54:34] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:54:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.48, 3.15 [14:58:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.33, 3.18 [14:59:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 8.88, 8.30 [15:00:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 4.00, 4.00 [15:00:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.75, 3.97 [15:02:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 4.00, 4.00 [15:03:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.81, 8.36, 8.31 [15:04:34] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:04:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.60, 3.84 [15:05:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 8.54, 8.06 [15:06:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.70, 3.80 [15:06:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.84, 4.24, 4.01 [15:07:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 8.33, 8.05 [15:07:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.72, 3.32 [15:07:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.23, 8.46, 8.43 [15:08:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.66, 3.77 [15:10:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.76, 4.25, 3.99 [15:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:16:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.68, 8.47 [15:18:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 8.90, 8.56 [15:20:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.81, 8.57 [15:20:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 8.96, 8.43 [15:22:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.13, 7.97, 8.27 [15:25:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.57, 3.92 [15:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:25:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 8.09, 8.27 [15:27:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.82, 3.96 [15:29:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.65, 3.90 [15:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:34:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 8.09, 8.08 [15:35:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.33, 1.85, 1.51 [15:35:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.18, 9.27, 8.68 [15:35:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.85, 8.95, 8.35 [15:36:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 9.56, 8.83 [15:36:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.03, 8.20, 8.13 [15:37:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 2.70, 3.32 [15:37:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.13, 8.92, 8.60 [15:39:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.21, 8.73 [15:41:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.87, 8.60, 8.59 [15:41:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.63, 8.15, 8.20 [15:42:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.34, 8.91 [15:44:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.33, 8.71, 8.75 [15:48:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.44, 3.96 [15:50:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.22, 8.78, 8.26 [15:52:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.67, 8.42, 8.20 [15:52:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.14, 7.84, 8.30 [15:53:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:56:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.25, 10.05, 9.29 [15:56:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.69, 9.94, 8.84 [15:56:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 3.28, 3.56 [15:57:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.55, 3.16 [15:58:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.37, 9.33, 9.12 [15:58:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.00, 3.42 [15:59:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 3.03, 3.01 [15:59:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 8.30, 7.95 [16:00:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.86, 9.17, 8.86 [16:00:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.40, 2.91, 3.32 [16:01:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.24, 8.20, 7.95 [16:02:04] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:07:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:08:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.59, 7.81, 8.42 [16:11:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.38, 9.52, 8.54 [16:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:13:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.55, 9.39, 8.63 [16:14:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.14, 1.64, 1.90 [16:16:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.53, 3.32 [16:17:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 8.38, 8.12 [16:17:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 8.40, 8.38 [16:18:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.98, 9.49, 9.02 [16:20:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.90, 9.47, 9.09 [16:20:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 3.15, 3.23 [16:20:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.17, 1.34, 1.68 [16:22:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.64, 7.65, 7.94 [16:26:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.76, 1.33, 0.95 [16:27:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.37, 7.67, 8.42 [16:28:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.44, 1.01, 0.87 [16:30:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.73, 3.43 [16:32:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.33, 3.31 [16:33:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 9.18, 8.82 [16:35:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.38, 8.23, 8.50 [16:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:38:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 8.79, 8.48 [16:39:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.51, 8.34, 8.53 [16:40:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.35, 9.06, 8.59 [16:41:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.49, 8.26, 8.49 [16:42:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.00, 3.32, 3.88 [16:42:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 8.19, 8.32 [16:45:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.43, 8.56, 8.62 [16:45:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:46:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.97, 8.26, 8.41 [16:47:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.53, 7.28, 8.16 [16:48:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.56, 4.20, 4.01 [16:50:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.27, 8.49, 8.52 [16:50:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.33, 3.76 [16:51:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.77, 8.54, 8.50 [16:51:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.22, 8.84, 8.15 [16:52:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.70, 9.22, 8.77 [16:54:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.34, 8.70, 8.63 [16:54:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.22, 8.79, 8.45 [16:54:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.84, 3.39 [16:55:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.84, 8.08, 8.41 [16:55:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.43, 7.88, 7.96 [16:56:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.81, 8.35, 8.36 [16:57:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.36, 8.05, 8.39 [16:58:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.95, 3.02, 3.17 [17:00:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.48, 3.88 [17:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:01:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 8.71, 8.20 [17:03:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 9.06, 8.39 [17:04:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.77, 3.88 [17:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:08:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.82, 3.89 [17:08:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.22, 1.65, 1.43 [17:09:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.28, 8.64 [17:11:03] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [17:11:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.36, 8.76 [17:12:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.58, 3.98 [17:12:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.55, 4.48, 4.09 [17:12:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 9.02, 8.62 [17:13:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.48, 8.35, 8.16 [17:14:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.95, 4.07 [17:15:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.65, 7.60, 7.91 [17:16:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.72, 8.21, 8.40 [17:17:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.35, 8.03, 8.43 [17:20:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.21, 3.51, 3.88 [17:21:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.33, 8.54 [17:21:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.45, 9.68, 9.03 [17:22:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.35, 8.53, 8.43 [17:22:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.48, 4.23, 4.09 [17:23:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.00, 8.90, 8.49 [17:23:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.32, 8.88, 8.82 [17:24:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.79, 9.46, 8.79 [17:25:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 9.50, 8.75 [17:27:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 8.95, 8.64 [17:29:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.64, 3.31 [17:29:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 9.46, 9.03 [17:29:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.51, 3.90 [17:29:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 9.33, 8.99 [17:30:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.71, 3.95 [17:31:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.70, 8.93, 8.90 [17:31:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.93, 4.00 [17:32:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.57, 4.64, 4.26 [17:33:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.13, 9.92, 9.14 [17:33:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.32, 3.23, 3.23 [17:33:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.92, 10.01, 9.11 [17:33:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.49, 3.82 [17:35:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 9.85, 9.23 [17:35:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.78, 3.96, 3.93 [17:35:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.46, 9.43, 9.06 [17:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.28, 8.79, 8.86 [17:41:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 9.68, 9.37 [17:42:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.19, 1.78, 1.96 [17:43:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 9.54, 9.34 [17:45:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 9.41, 9.32 [17:45:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.78, 7.44, 8.31 [17:48:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.09, 1.32, 1.68 [17:49:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.38, 9.88, 9.05 [17:49:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.39, 9.17, 8.88 [17:49:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.75, 7.76, 8.35 [17:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.60, 9.70, 9.09 [17:51:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:51:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 8.90, 8.81 [17:53:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 8.34, 8.43 [17:55:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 8.31, 8.42 [17:57:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 9.71, 9.23 [17:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.61, 9.74, 9.11 [17:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.84, 9.49, 9.11 [17:59:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 9.02, 8.71 [18:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:00:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 8.96, 8.77 [18:03:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.24, 9.91, 9.56 [18:03:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.32, 7.92, 8.34 [18:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.11, 9.66, 9.25 [18:06:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.89, 9.26 [18:07:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.17, 9.23, 8.78 [18:08:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.66, 3.08, 3.93 [18:09:02] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.94, 1.62, 1.52 [18:09:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 9.56, 8.98 [18:10:11] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [18:10:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:10:56] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.77, 1.59 [18:11:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.16, 8.92, 9.00 [18:11:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.22, 9.45, 9.33 [18:11:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.86, 10.01, 9.21 [18:13:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.56, 9.05, 9.04 [18:13:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.78, 8.95, 8.91 [18:14:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.23, 3.27 [18:15:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.07, 9.60, 9.14 [18:16:54] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/.github/commit/deb46d9fa4e3 [18:16:55] [02miraheze/.github] 07Universal-Omega 03deb46d9 - Rename workflow-templates/.github/workflows/mediawiki-tests.yml to workflow-templates/action.yml [18:16:57] [02.github] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/.github [18:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:17:09] [02.github] 07Universal-Omega opened pull request 03#8: Rename workflow-templates/.github/workflows/mediawiki-tests.yml to workflow-templates/action.yml - 13https://github.com/miraheze/.github/pull/8 [18:17:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.96, 7.33, 8.40 [18:18:16] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/.github/compare/deb46d9fa4e3...b190188a3e97 [18:18:17] [02miraheze/.github] 07Universal-Omega 03b190188 - Update sync-tests.yml [18:18:19] [02.github] 07Universal-Omega synchronize pull request 03#8: Rename workflow-templates/.github/workflows/mediawiki-tests.yml to workflow-templates/action.yml - 13https://github.com/miraheze/.github/pull/8 [18:18:43] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4f9fbf8f9751...d0ae79843d68 [18:18:44] [02miraheze/CreateWiki] 07JohnFLewis 03d0ae798 - T9018: Add submission check for anonymous users POSTing [18:19:02] [02.github] 07Universal-Omega closed pull request 03#8: Rename workflow-templates/.github/workflows/mediawiki-tests.yml to workflow-templates/action.yml - 13https://github.com/miraheze/.github/pull/8 [18:19:03] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/.github/compare/a5cf26ab8c40...a965ce2b95a2 [18:19:04] [02miraheze/.github] 07Universal-Omega 03a965ce2 - Rename workflow-templates/.github/workflows/mediawiki-tests.yml to workflow-templates/action.yml (#8) [18:19:06] [02.github] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/.github [18:19:07] [02miraheze/.github] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:19:19] [02miraheze/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/workflow-templates/rectalogic/notify-irc-2 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/b7d5ceacf925 [18:19:20] [02miraheze/.github] 07dependabot[bot] 03b7d5cea - Bump rectalogic/notify-irc from 1 to 2 in /workflow-templates [18:19:22] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/workflow-templates/rectalogic/notify-irc-2 - 13https://github.com/miraheze/.github [18:19:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.78, 10.24, 9.42 [18:19:23] [02.github] 07dependabot[bot] opened pull request 03#9: Bump rectalogic/notify-irc from 1 to 2 in /workflow-templates - 13https://github.com/miraheze/.github/pull/9 [18:19:25] [02.github] 07dependabot[bot] labeled pull request 03#9: Bump rectalogic/notify-irc from 1 to 2 in /workflow-templates - 13https://github.com/miraheze/.github/pull/9 [18:19:50] [02.github] 07Universal-Omega closed pull request 03#9: Bump rectalogic/notify-irc from 1 to 2 in /workflow-templates - 13https://github.com/miraheze/.github/pull/9 [18:19:52] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/a965ce2b95a2...aac18df32f0c [18:19:53] [02miraheze/.github] 07dependabot[bot] 03aac18df - Bump rectalogic/notify-irc from 1 to 2 in /workflow-templates (#9) [18:19:55] [02miraheze/.github] 07Universal-Omega deleted branch 03dependabot/github_actions/workflow-templates/rectalogic/notify-irc-2 [18:19:56] [02.github] 07Universal-Omega deleted branch 03dependabot/github_actions/workflow-templates/rectalogic/notify-irc-2 - 13https://github.com/miraheze/.github [18:20:00] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/091931358bc5...1d6d067894df [18:20:01] [02miraheze/MatomoAnalytics] 07Universal-Omega 031d6d067 - Update workflow [18:20:02] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/7ab565fbf191...f43433cce820 [18:20:04] [02miraheze/MirahezeMagic] 07Universal-Omega 03f43433c - Update workflow [18:20:06] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/ec357ab46f78...39ef02187646 [18:20:07] [02miraheze/PDFEmbed] 07Universal-Omega 0339ef021 - Update workflow [18:20:22] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/e0680542e3a8...5d3c650e7202 [18:20:23] [02miraheze/RemovePII] 07Universal-Omega 035d3c650 - Update workflow [18:20:41] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/5d3589b84ab1...3c4a633cb554 [18:20:42] [02miraheze/RottenLinks] 07Universal-Omega 033c4a633 - Update workflow [18:20:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.54, 3.92 [18:20:49] [02miraheze/SpriteSheet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/ebeda3312a3f...3c16b75e80b1 [18:20:50] [02miraheze/SpriteSheet] 07Universal-Omega 033c16b75 - Update workflow [18:20:56] [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/YouTube/compare/3e15254b1858...2c4c7037367a [18:20:57] [02miraheze/YouTube] 07Universal-Omega 032c4c703 - Update workflow [18:20:59] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/9da3832cc620...bbaf03ffa7da [18:21:00] [02miraheze/WikiDiscover] 07Universal-Omega 03bbaf03f - Update workflow [18:21:28] miraheze/CreateWiki - JohnFLewis the build has errored. [18:21:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 9.30, 9.20 [18:22:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:23:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.27, 9.77, 9.47 [18:24:26] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/32dc7f509d41...3035606a0e09 [18:24:27] [02miraheze/mediawiki] 07JohnFLewis 033035606 - update CreateWiki [18:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.97, 8.54, 8.53 [18:26:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.33, 9.21, 8.75 [18:28:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 9.32, 8.86 [18:28:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.76, 7.69, 8.27 [18:28:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 2.62, 3.25 [18:32:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.14, 8.14, 8.43 [18:33:14] !log [johnflewis@mwtask111] starting deploy of {'world': True} to all [18:33:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.39, 9.12, 9.19 [18:33:34] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:35:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.11, 9.17 [18:36:17] [02miraheze/CustomHeader] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CustomHeader/compare/ba41ba693e80...2dc33e3ddbab [18:36:19] [02miraheze/CustomHeader] 07Universal-Omega 032dc33e3 - Update workflow [18:36:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.58, 8.79, 8.68 [18:36:41] miraheze/MatomoAnalytics - Universal-Omega the build passed. [18:36:44] miraheze/PDFEmbed - Universal-Omega the build passed. [18:36:53] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d0ae79843d68...68d55912917b [18:36:55] [02miraheze/CreateWiki] 07Universal-Omega 0368d5591 - Update workflow [18:37:07] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/22880cc40481...4b61f0e5be98 [18:37:08] [02miraheze/DataDump] 07Universal-Omega 034b61f0e - Update workflow [18:37:17] miraheze/RemovePII - Universal-Omega the build passed. [18:37:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.06, 9.14, 9.15 [18:37:34] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/94a76e414286...cec03ce09637 [18:37:35] [02miraheze/GlobalNewFiles] 07Universal-Omega 03cec03ce - Update workflow [18:37:55] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/be76a9139d17...6df0c6572162 [18:37:57] [02miraheze/IncidentReporting] 07Universal-Omega 036df0c65 - Update workflow [18:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:38:45] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/8a3a98bf0ec1...740bb6eff4e3 [18:38:47] [02miraheze/ManageWiki] 07Universal-Omega 03740bb6e - Update workflow [18:38:49] !log [johnflewis@test101] starting deploy of {'world': True} to all [18:39:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:05] miraheze/SpriteSheet - Universal-Omega the build passed. [18:39:10] miraheze/YouTube - Universal-Omega the build passed. [18:39:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.35, 8.43, 8.89 [18:39:36] miraheze/RottenLinks - Universal-Omega the build passed. [18:39:44] miraheze/WikiDiscover - Universal-Omega the build passed. [18:40:33] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:41:15] !log [johnflewis@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 481s [18:41:16] miraheze/CreateWiki - Universal-Omega the build has errored. [18:41:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.80, 8.60, 8.43 [18:41:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:41:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.66, 10.26, 9.38 [18:43:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.01, 8.49, 8.41 [18:43:33] !log [johnflewis@test101] finished deploy of {'world': True} to all - SUCCESS in 284s [18:43:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.70, 9.52, 9.21 [18:44:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.30, 8.00, 8.41 [18:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:47:36] miraheze/CustomHeader - Universal-Omega the build passed. [18:48:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.29, 3.11 [18:48:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 3.56, 3.20 [18:49:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 8.67, 8.46 [18:49:19] miraheze/DataDump - Universal-Omega the build passed. [18:49:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.95, 10.26, 9.46 [18:50:03] miraheze/GlobalNewFiles - Universal-Omega the build passed. [18:50:09] miraheze/IncidentReporting - Universal-Omega the build passed. [18:50:28] miraheze/ManageWiki - Universal-Omega the build passed. [18:50:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 2.98, 3.02 [18:50:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.63, 3.27 [18:51:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.23, 9.11, 8.64 [18:51:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.50, 9.58, 8.95 [18:51:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.55, 9.23 [18:52:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 3.19, 3.14 [18:52:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 9.46, 8.83 [18:53:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 9.75, 9.46 [18:55:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.78, 10.16, 9.62 [18:55:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.01, 9.15, 9.19 [18:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.00, 9.11, 9.29 [18:58:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.33, 3.19 [18:59:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.56, 8.78, 9.00 [19:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:00:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.56, 3.28 [19:01:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.44, 9.59, 9.27 [19:01:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.84, 3.38 [19:02:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.29, 8.18, 8.48 [19:02:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.54, 3.31 [19:03:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.12, 8.86, 9.06 [19:03:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.43, 2.19, 1.41 [19:03:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.44, 3.12 [19:04:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 3.34, 3.28 [19:05:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.21, 9.36, 9.21 [19:05:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.55, 1.26 [19:06:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 8.80, 8.69 [19:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 9.69, 9.36 [19:07:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.97, 3.27, 3.13 [19:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:08:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.78, 3.50 [19:09:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.57, 9.80, 9.43 [19:09:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 8.72, 8.67 [19:12:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.19, 3.35 [19:12:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.73, 9.53, 8.95 [19:13:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:13:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.45, 3.66 [19:13:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.14, 8.49, 8.66 [19:14:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.76, 8.42, 8.61 [19:14:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.15, 9.45, 9.52 [19:15:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.08, 7.80, 8.40 [19:16:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.96, 3.65 [19:17:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.79, 7.57, 8.38 [19:18:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.97, 3.68 [19:20:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.86, 3.67 [19:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:22:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 4.14, 3.79 [19:23:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 4.08, 3.80 [19:26:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.47, 3.60 [19:27:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.41, 3.60 [19:27:10] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/68d55912917b...d7270203c535 [19:27:11] [02miraheze/CreateWiki] 07Universal-Omega 03d727020 - Fix CI [19:28:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.34, 4.44, 3.94 [19:28:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.83, 8.00, 8.49 [19:29:15] .tell CosmicAlpha in future version we might not be able to use isRegistered [19:29:16] RhinosF1: I'll pass that on when CosmicAlpha is around. [19:29:27] Cc JohnLewis [19:29:48] hm? [19:30:13] JohnLewis: look at UO's commit [19:30:33] Yeah, I don't understand why the change was necessary [19:30:54] something about CI [19:30:56] or if that's the best change [19:31:10] But we need to be careful when the temp account nonsense comes in for anons [19:31:13] do we really want to disallow anonymous users from viewing Special:RequestWikiQueue? [19:31:22] There's got to be a less blunt way to fix this [19:31:38] dmehus: go away [19:31:42] that doesn't do that [19:31:47] RhinosF1: yeah, but the CI failure is irrelevant to the change he's done [19:31:52] The method is called submitForm [19:31:56] RhinosF1: why "go away"? [19:31:57] JohnLewis: right [19:32:14] dmehus: because you have 0 idea what you're talking about and spewing words [19:32:18] Can you please explain to me what the change is doing then? [19:32:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.39, 8.60, 8.62 [19:32:49] dmehus: checks you are a registered user when submitting the form on a request to make a comment [19:32:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 2.78, 3.27 [19:33:16] RhinosF1, ack, thanks. Please next time just tell me this. I'm satisfied with that ;) [19:34:16] dmehus: it's annoying. You make a lot of comments that make no sense with 0 technical knowledge [19:34:25] It would have taken seconds to look at the method [19:34:33] If you want to be involved, make an effort [19:34:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.21, 8.48 [19:36:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 9.53, 8.96 [19:39:05] miraheze/CreateWiki - Universal-Omega the build passed. [19:40:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 8.97, 8.87 [19:40:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.11, 8.35, 8.61 [19:42:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 9.05, 8.52 [19:42:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.38, 8.63, 8.78 [19:45:16] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2b86e3773774...1b0c0b281f98 [19:45:18] [02miraheze/puppet] 07JohnFLewis 031b0c0b2 - db: re-enablee db backups [19:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:45:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.69, 3.43 [19:45:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.88, 8.38, 8.37 [19:46:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.16, 8.36, 8.05 [19:47:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.98, 8.85, 8.24 [19:50:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.61, 3.95 [19:50:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.08, 7.34, 8.14 [19:51:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 9.33, 8.66 [19:52:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.45, 3.96, 4.06 [19:55:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.83, 3.56 [19:55:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:55:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 9.60, 8.89 [19:57:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.96, 3.66 [19:57:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.30, 8.83, 8.68 [19:58:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.74, 8.38, 8.29 [20:01:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.57, 9.39, 8.91 [20:02:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.69, 7.89, 8.47 [20:03:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.07, 8.80, 8.75 [20:04:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.43, 8.45, 8.39 [20:05:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.58, 9.05 [20:06:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.39, 2.02, 1.45 [20:06:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.13, 8.68, 8.39 [20:06:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.39, 9.12, 8.84 [20:07:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.87, 3.27 [20:07:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.47, 9.07 [20:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:08:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.65, 1.38 [20:08:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.15, 8.36, 8.60 [20:08:26] PROBLEM - olwest.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for olwest.org could not be found [20:09:37] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.68, 9.19 [20:11:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.43, 3.28 [20:11:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.62, 8.65, 8.86 [20:12:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.86, 8.50, 8.45 [20:12:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.78, 8.08, 8.46 [20:13:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.67, 3.37 [20:13:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.11, 3.69 [20:15:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.76, 1.48 [20:15:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 8.78, 8.58 [20:17:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.44, 1.25, 1.32 [20:17:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.41, 8.24, 8.40 [20:18:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:18:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.27, 3.85 [20:19:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.73, 3.56 [20:21:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.89, 3.99 [20:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:25:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 3.03, 3.31 [20:25:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.54, 9.97, 9.23 [20:26:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.25, 8.91, 8.61 [20:26:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 2.57, 3.32 [20:27:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.36, 9.14, 9.01 [20:27:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 8.87, 8.63 [20:28:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.53, 9.30, 8.76 [20:31:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 8.60, 8.35 [20:32:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.24, 9.09, 8.85 [20:33:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 8.94, 8.64 [20:33:45] RhinosF1: that’s rude of you to tell Doug to go away [20:33:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.47, 8.14, 8.21 [20:33:58] Also very hypocritical of yourself as well [20:35:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.19, 8.78 [20:35:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.49, 3.57 [20:36:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.73, 9.55, 9.04 [20:37:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.33, 3.51 [20:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 9.14, 8.96 [20:38:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.51, 3.37 [20:39:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.94, 3.73 [20:40:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.94, 3.53 [20:41:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.32, 9.01, 8.92 [20:42:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.74, 3.51 [20:43:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:43:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 8.96, 8.91 [20:44:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 4.39, 3.79 [20:44:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.82, 7.76, 8.33 [20:46:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.56, 3.33 [20:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:48:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.37, 3.30 [20:49:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.47, 7.94, 8.42 [20:50:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.73, 3.67 [20:51:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.89, 8.08, 8.49 [20:55:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.10, 8.80 [20:55:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.49, 9.32, 8.84 [20:56:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.93, 4.01, 3.80 [20:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 9.23, 8.90 [20:57:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.65, 3.50, 3.30 [20:59:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.72, 8.96, 8.88 [21:03:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.00, 7.85, 8.42 [21:03:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.40, 8.91, 8.33 [21:03:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.75, 3.56 [21:03:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.42, 7.68, 8.34 [21:05:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 9.14, 8.49 [21:06:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 8.13, 7.64 [21:07:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.18, 8.46, 8.56 [21:07:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.99, 8.29, 8.25 [21:07:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.03, 9.60, 9.01 [21:08:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.76, 7.72, 7.54 [21:09:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.04, 8.03, 8.40 [21:09:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 3.08, 3.33 [21:11:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.94, 9.02, 8.91 [21:15:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.39, 3.39 [21:15:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.54, 1.11 [21:17:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.05, 1.40, 1.11 [21:17:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.65, 7.40, 8.23 [21:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:19:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.23, 3.36 [21:21:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.57, 1.93, 1.35 [21:22:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.60, 1.30 [21:23:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.62, 1.31 [21:24:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.40, 1.26 [21:24:33] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:26:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.00, 3.24, 3.97 [21:28:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.43, 3.40 [21:30:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 2.93, 3.22 [21:31:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.35, 8.51, 8.15 [21:33:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.45, 8.30, 8.11 [21:36:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.03, 8.51, 8.17 [21:38:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.41, 7.90, 7.97 [21:38:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.88, 2.69, 3.34 [21:39:33] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:41:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.47, 3.35 [21:41:56] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [21:42:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.19, 3.81 [21:43:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.84, 3.49 [21:44:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.92, 3.76 [21:45:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.35, 3.36 [21:46:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.98, 3.79 [21:46:52] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 1035 MB (10% inode=80%); [21:47:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:51:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.17, 3.48, 3.98 [21:52:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 8.47, 8.18 [21:52:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.51, 3.68 [21:54:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.46, 7.77, 7.96 [21:55:51] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.08, 7.54, 5.31 [21:57:02] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:57:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 4.06, 4.00 [21:58:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.62, 8.24, 7.88 [21:59:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.92, 3.95 [22:00:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 8.09, 7.87 [22:01:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 3.93, 3.93 [22:04:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.45, 9.13, 8.39 [22:05:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.22, 8.62, 8.26 [22:06:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 8.94, 8.42 [22:06:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.83, 2.81, 3.27 [22:07:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 8.96, 8.41 [22:09:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 9.07, 8.52 [22:10:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.59, 8.33, 8.31 [22:11:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 8.38, 8.33 [22:11:30] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [22:11:30] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:13:24] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.39, 5.32, 5.90 [22:14:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.05, 8.42 [22:15:20] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.75, 5.77, 5.94 [22:15:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 9.17, 8.69 [22:16:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.26, 8.40, 8.26 [22:16:58] PROBLEM - olwest.org - LetsEncrypt on sslhost is CRITICAL: connect to address olwest.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:17:17] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.55, 4.76, 5.55 [22:19:14] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.19, 5.92, 5.88 [22:20:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.98, 9.11, 8.60 [22:21:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.45, 8.87, 8.62 [22:23:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.43, 8.93, 8.70 [22:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:26:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.58, 3.39 [22:27:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.59, 7.82, 8.29 [22:28:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.58, 3.39 [22:29:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.04, 7.61, 8.22 [22:30:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.23, 3.29 [22:30:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.29, 1.54, 1.94 [22:33:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.34, 3.89 [22:35:03] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.09, 4.84, 5.94 [22:36:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:36:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 8.40, 8.06 [22:36:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.11, 1.27, 1.68 [22:38:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.60, 8.02, 7.96 [22:41:03] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.50, 3.75, 5.08 [22:41:04] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:45:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 2.85, 3.34 [22:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:58:52] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14149 MB (10% inode=99%); [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:04] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:52] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 16166 MB (12% inode=99%); [23:01:51] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.04, 5.48, 4.77 [23:03:48] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.93, 5.00, 4.68 [23:05:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 8.31, 7.62 [23:07:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.82, 7.71, 7.48 [23:10:56] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.45, 2.21, 1.68 [23:11:17] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 675s [23:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:12:42] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 758s [23:12:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 3.96, 3.24 [23:12:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.63, 3.20 [23:16:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.24, 3.10 [23:18:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:23:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:24:11] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.75, 5.88, 4.87 [23:24:52] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13511 MB (10% inode=99%); [23:26:52] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 43328 MB (32% inode=99%); [23:27:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.93, 9.16, 8.06 [23:30:02] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.37, 5.86, 5.25 [23:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:30:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 3.94, 3.38 [23:31:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.51, 3.08 [23:32:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.65, 3.34 [23:33:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.09, 8.63, 7.72 [23:33:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.18, 3.02 [23:33:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 8.26, 8.00 [23:33:56] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.76, 7.03, 5.74 [23:34:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 3.34, 3.27 [23:35:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 9.10, 7.99 [23:35:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 9.18, 8.05 [23:35:53] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.35, 5.60, 5.39 [23:37:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 8.65, 8.19 [23:37:50] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.02, 4.73, 5.09 [23:38:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.49, 8.46, 7.99 [23:41:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.29, 3.67, 3.22 [23:41:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.24, 8.50, 8.25 [23:42:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.73, 9.01, 7.23 [23:42:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 7.97, 7.97 [23:42:58] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:01] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.42, 3.18 [23:43:19] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:23] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:38] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 5.79, 6.31, 5.73 [23:43:40] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:07] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.60, 8.71, 7.36 [23:44:29] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 4.22, 3.65 [23:45:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.28, 3.16 [23:45:35] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.41, 5.73, 5.59 [23:45:49] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:46:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.32, 7.46, 7.07 [23:46:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 9.47, 8.54 [23:46:30] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:46:40] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:46:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.98, 3.64 [23:46:53] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:46:57] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:46:58] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:47:45] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:47:51] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [23:47:59] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 23 minutes ago with 0 failures [23:48:14] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.033 second response time [23:48:30] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [23:48:36] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [23:49:01] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:49:02] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [23:49:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.84, 6.24, 5.79 [23:50:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.74, 7.96, 8.13 [23:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:50:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 4.42, 3.87 [23:51:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.34, 5.02, 5.39 [23:52:13] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:38] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:52] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 3.62, 3.70 [23:54:46] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:54:49] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:55:07] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:55:07] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:55:19] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.87, 4.07, 4.92 [23:55:50] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:55:51] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:55:52] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:56:05] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:56:06] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:56:09] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 1.563 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [23:56:12] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.367 second response time [23:56:19] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [23:56:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 8.45, 8.22 [23:56:45] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 24292 MB (36% inode=89%); [23:56:47] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 2.366 second response time [23:56:57] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 3.025 second response time [23:59:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.00, 4.25, 3.47