[00:03:10] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 2218 MB (23% inode=80%); [00:03:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.55, 1.27 [00:03:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.36, 8.42, 7.51 [00:03:57] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.59, 2.94 [00:04:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.50, 3.06 [00:04:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.14, 4.07, 3.05 [00:05:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.31, 1.56, 1.31 [00:05:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.90, 9.25, 7.94 [00:06:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.59, 3.04 [00:06:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.45, 3.10 [00:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 8.95, 8.09 [00:06:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.77, 3.07 [00:07:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.94, 8.40, 7.76 [00:08:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 3.28, 2.97 [00:10:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.34, 3.13 [00:10:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.27, 7.69, 7.78 [00:11:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.02, 8.84, 8.05 [00:12:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.23, 3.08 [00:12:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 8.56, 7.77 [00:13:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.74, 8.54, 8.05 [00:14:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.08, 8.06, 7.69 [00:15:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.86, 8.23, 7.98 [00:17:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 4.07, 3.52 [00:19:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.90, 3.51 [00:20:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 8.93, 7.92 [00:20:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.76, 4.24, 3.68 [00:23:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 10.01, 8.76 [00:24:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.40, 3.48 [00:25:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 9.70, 8.80 [00:26:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.42, 8.62, 8.17 [00:26:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.12, 3.36 [00:28:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.25, 9.45, 8.54 [00:29:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.46, 9.62, 8.93 [00:30:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.04, 9.10, 8.53 [00:31:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.43, 9.34, 8.90 [00:31:57] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 8.23, 8.27 [00:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:35:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 9.33, 8.50 [00:35:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.70, 9.25, 8.68 [00:37:52] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:38:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.97, 1.84, 1.52 [00:38:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.45, 8.39, 8.28 [00:39:52] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.71, 8.34, 8.44 [00:42:53] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.50, 1.70, 1.54 [00:42:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 8.88, 8.51 [00:43:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.52, 8.17, 8.48 [00:44:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.63, 8.20, 8.31 [00:45:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:47:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.01, 10.20, 9.14 [00:49:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 9.70, 8.94 [00:51:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.19, 9.92, 9.34 [00:52:52] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.49, 1.95, 1.69 [00:54:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 8.64, 8.39 [00:54:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.94, 1.72 [00:55:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.10, 10.10, 9.52 [00:56:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.23, 1.66, 1.65 [00:58:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.11, 8.01, 8.21 [00:59:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.98, 10.00, 9.59 [00:59:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.41, 3.09 [01:01:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.80, 9.43, 9.50 [01:01:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.57, 3.19 [01:03:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.33, 3.32, 3.13 [01:04:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.97, 4.53, 3.64 [01:04:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 8.80, 8.40 [01:05:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:06:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.39, 7.90, 8.12 [01:07:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.58, 3.29 [01:07:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [01:08:12] I'v deleted my secont comment which was invalid. [01:09:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.35, 4.44, 3.65 [01:10:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.45, 8.59, 7.85 [01:11:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.42, 3.16 [01:11:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.33, 9.37 [01:12:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.96, 3.83 [01:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.12, 3.22, 3.11 [01:13:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.24, 8.58, 9.09 [01:14:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 4.35, 4.01 [01:14:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 8.81, 8.12 [01:16:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.72, 3.82 [01:16:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 8.54, 8.32 [01:16:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.90, 3.96, 2.24 [01:16:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 8.45, 8.07 [01:17:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.66, 9.37, 9.24 [01:17:33] *second [01:18:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.73, 8.21, 8.22 [01:18:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 4.00, 3.79 [01:19:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.91, 9.48, 9.04 [01:20:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 8.70, 8.27 [01:21:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 8.87, 8.87 [01:21:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.79, 9.00, 9.18 [01:22:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.34, 8.06, 8.08 [01:24:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.68, 3.62 [01:24:30] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.83, 1.61 [01:25:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 7.82, 8.44 [01:26:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.72, 1.90 [01:26:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.52, 3.58 [01:26:28] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.20, 1.60, 1.55 [01:27:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.82, 9.96, 9.44 [01:28:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.87, 4.27, 3.87 [01:30:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.78, 3.63 [01:30:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 4.00, 3.45 [01:31:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 9.42, 9.33 [01:32:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.32, 1.66 [01:32:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.99, 3.52 [01:35:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.65, 8.48, 8.11 [01:37:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.87, 7.67, 7.87 [01:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:44:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:44:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 3.10, 3.32 [01:45:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.42, 8.75, 8.81 [01:45:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 7.86, 7.65 [01:47:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.97, 8.04, 8.55 [01:47:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.38, 7.54, 7.55 [01:49:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.25, 9.04, 8.84 [01:51:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.52, 8.77, 8.77 [01:52:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.25, 3.14, 3.87 [01:52:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.64, 3.97 [01:54:01] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.11, 1.85, 1.57 [01:55:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.30, 1.62, 1.52 [01:58:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.58, 3.79 [02:00:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.49, 3.73 [02:02:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.50, 4.05, 3.92 [02:04:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.57, 3.76 [02:04:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 2.58, 3.27 [02:04:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.65, 2.01, 1.34 [02:05:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.04, 9.27, 8.82 [02:06:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.58, 3.72 [02:06:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.30, 1.70, 1.31 [02:06:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.05, 9.07, 8.41 [02:06:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.55, 8.48, 7.87 [02:07:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 8.95, 8.76 [02:08:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.35, 8.58 [02:10:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.58, 8.96, 8.53 [02:10:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.61, 7.69, 7.77 [02:12:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.52, 3.71 [02:12:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.53, 9.25, 8.17 [02:12:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.20, 7.80, 8.18 [02:13:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.79, 8.02, 8.44 [02:14:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 8.74, 8.11 [02:16:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 8.23, 8.00 [02:18:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 2.61, 3.28 [02:18:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 8.93, 8.44 [02:19:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:20:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.10, 8.82, 8.65 [02:22:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.69, 4.73, 4.04 [02:22:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.76, 8.09, 8.40 [02:22:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.26, 3.16 [02:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:22:33] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.74, 1.95, 1.63 [02:22:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.28, 8.41, 8.37 [02:24:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.04, 3.09 [02:26:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.92, 3.87 [02:26:29] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.40, 1.93, 1.72 [02:26:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 8.62, 8.49 [02:28:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.71, 4.21 [02:28:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.58, 3.27 [02:28:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.11, 9.17, 8.70 [02:30:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.70, 3.37 [02:30:26] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.43, 1.67, 1.67 [02:30:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 9.32, 8.82 [02:34:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 4.36, 3.67 [02:34:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.65, 7.97, 8.38 [02:36:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 8.65, 7.90 [02:37:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.93, 1.58 [02:38:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.64, 3.57 [02:38:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.21, 8.25, 7.83 [02:39:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.70, 1.54 [02:44:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.17, 3.38 [02:46:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.00, 8.30, 8.18 [02:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:50:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.86, 8.34, 8.24 [02:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.50, 3.95 [03:04:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.52, 3.82 [03:04:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.72, 9.40, 8.44 [03:06:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.83, 3.92 [03:06:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.73, 8.98, 8.39 [03:08:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 3.97, 3.93 [03:08:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.41, 8.05, 8.12 [03:13:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:14:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.50, 3.81 [03:16:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/c9d14d7b2499...4360bd78c28b [03:16:08] [02miraheze/CreateWiki] 07Universal-Omega 034360bd7 - Update CreateWikiRegexConstraint.php [03:16:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [03:18:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.83, 8.71, 8.14 [03:19:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.57, 3.35 [03:20:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.68, 3.71 [03:23:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 4.15, 3.64 [03:23:36] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.24, 1.69, 1.42 [03:23:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.50, 3.02 [03:24:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.89, 3.79 [03:25:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.49, 3.46 [03:25:24] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4360bd78c28b...a9d4d7ecf760 [03:25:25] [02miraheze/CreateWiki] 07Universal-Omega 03a9d4d7e - Update CreateWikiRegexConstraint.php [03:25:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 9.34, 8.53 [03:25:27] [02CreateWiki] 07Universal-Omega synchronize pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [03:25:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.04, 2.90 [03:25:35] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.31, 1.55, 1.40 [03:25:48] [02CreateWiki] 07Universal-Omega reviewed pull request 03#312 commit - 13https://github.com/miraheze/CreateWiki/pull/312#discussion_r835705914 [03:25:59] [02CreateWiki] 07Universal-Omega reviewed pull request 03#312 commit - 13https://github.com/miraheze/CreateWiki/pull/312#discussion_r835705925 [03:26:13] [02CreateWiki] 07Universal-Omega reviewed pull request 03#312 commit - 13https://github.com/miraheze/CreateWiki/pull/312#discussion_r835705979 [03:26:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.44, 8.21, 8.18 [03:28:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.33, 4.19, 3.86 [03:28:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.95, 3.65 [03:29:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.34, 3.71, 3.25 [03:30:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.73, 3.73 [03:30:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.37, 3.47 [03:31:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.27, 3.14 [03:31:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.34, 8.38, 8.43 [03:32:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.69, 3.58 [03:33:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:36:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.66, 3.60 [03:37:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:33] miraheze/CreateWiki - Universal-Omega the build passed. [03:38:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 2.94, 3.34 [03:41:43] .in 9h 50m renew ssl certs [03:41:43] MacFan4000: Okay, will remind at 2022-03-26 - 09:31:43EDT [03:42:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.03, 3.37 [03:45:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.01, 8.66, 7.86 [03:47:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.75, 8.26, 7.81 [03:49:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 8.29, 7.76 [03:52:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.78, 3.53 [03:53:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 8.44, 7.97 [03:54:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.32, 3.38 [03:55:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 8.14, 7.92 [03:57:12] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.81, 8.44, 8.12 [03:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:05:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 8.35, 7.99 [04:06:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 8.72, 7.88 [04:07:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 8.32, 8.04 [04:08:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.86, 8.28, 7.81 [04:08:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.77, 3.59, 3.02 [04:10:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.31, 2.99 [04:13:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.17, 8.24, 7.65 [04:15:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.21, 7.40, 7.40 [04:18:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.62, 3.20 [04:18:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.27, 2.96 [04:20:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 2.93, 2.87 [04:21:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.81, 3.44 [04:22:27] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [04:22:27] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40574312210083 seconds [04:22:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.76, 2.95, 3.03 [04:23:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.28, 3.28 [04:23:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:23:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 8.32, 8.13 [04:25:37] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.71, 7.91, 8.02 [04:26:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:29:12] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/p [04:29:12] ist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404481649398804 seconds [04:34:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 4.32, 3.66 [04:34:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 3.65, 3.29 [04:36:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.93, 3.61 [04:36:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.98, 3.22, 3.17 [04:40:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.35, 4.23, 3.84 [04:41:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:42:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.67, 3.68 [04:46:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.90, 3.38 [04:50:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.61, 3.57 [04:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:52] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:52:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.45, 3.22 [04:54:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.24, 3.17 [04:56:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.66, 3.57 [04:58:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.63, 4.59, 3.74 [05:02:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.61, 3.72 [05:02:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.86, 3.64 [05:05:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:08:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.46, 3.46 [05:10:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.91, 3.49, 3.51 [05:10:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.58, 3.51 [05:12:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 4.19, 3.75 [05:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:16:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.47, 3.59 [05:16:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.90, 3.27, 3.51 [05:19:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:20:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.07, 2.79, 3.26 [05:22:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.02, 3.35 [05:28:10] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:30:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:30:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.88, 3.55 [05:32:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.95, 3.60 [05:34:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.54, 3.48 [05:36:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.95, 3.66 [05:38:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.61, 3.57 [05:42:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.72, 3.22 [05:45:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:52:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:52] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.06, 1.60, 1.12 [06:24:53] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.20, 1.48, 1.14 [06:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:53:14] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a9d4d7ecf760...6a6c075731bf [06:53:16] [02miraheze/CreateWiki] 07Universal-Omega 036a6c075 - Update CreateWikiRegexConstraint.php [06:53:17] [02CreateWiki] 07Universal-Omega synchronize pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [06:55:05] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.56, 9.82, 6.84 [06:57:02] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.74, 7.28, 6.25 [06:58:40] [02CreateWiki] 07lens0021 commented on pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312#issuecomment-1079624951 [06:59:00] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.34, 6.47, 6.09 [07:00:42] [02CreateWiki] 07Universal-Omega commented on pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312#issuecomment-1079625296 [07:02:52] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.42, 8.53, 7.07 [07:05:15] miraheze/CreateWiki - Universal-Omega the build passed. [07:18:32] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.26, 6.68, 7.64 [07:20:29] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 13.53, 13.09, 10.18 [07:36:09] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.72, 6.45, 7.92 [07:42:01] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.11, 4.68, 6.61 [07:43:07] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [07:43:07] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:51:45] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 11.37, 7.35, 6.49 [07:57:37] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.61, 7.62, 7.23 [07:59:35] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.08, 6.17, 6.75 [08:11:34] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 12.77, 10.43, 7.61 [08:12:40] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [08:15:34] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.78, 7.12, 6.88 [08:15:58] !log INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); (eznirwiki) [08:16:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:17:34] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.66, 6.18, 6.55 [08:18:27] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=eznirwiki (END - exit=0) [08:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:18:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.21, 2.50 [08:19:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:20:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.77, 2.78 [08:22:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.47, 2.78 [08:23:34] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.71, 7.90, 7.18 [08:24:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:24:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.19, 2.76 [08:25:34] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 12.27, 10.42, 8.23 [08:27:32] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.70, 2.37, 1.44 [08:29:28] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.07, 1.79, 1.33 [08:31:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.72, 1.48, 1.28 [08:36:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.70, 3.13 [08:41:19] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:42:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.86, 3.39 [08:46:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.69, 2.80, 3.07 [08:54:11] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=numberpediawiki --update (END - exit=0) [08:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.52, 3.44, 1.86 [09:02:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 4.21, 3.27 [09:02:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.14, 3.84, 2.80 [09:02:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.53, 1.76, 1.51 [09:04:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.20, 1.23, 1.35 [09:06:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.64, 2.98 [09:07:34] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.84, 5.61, 7.63 [09:09:19] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [09:09:34] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 0.96, 4.02, 6.80 [09:10:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:13:42] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:13:42] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:22:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.96, 3.11 [09:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:30:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.94, 3.48 [09:32:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.34, 3.33 [09:36:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.50, 3.94 [09:42:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 4.05, 4.00 [09:43:15] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:46:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.56, 3.26 [09:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 8.40, 7.45 [09:50:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.71, 3.89 [09:50:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.08, 3.30, 3.22 [09:51:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 8.68, 7.65 [09:53:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.26, 8.20, 7.60 [09:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:57:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.34, 8.57, 7.92 [09:58:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 3.93, 3.84 [09:59:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.54, 7.75, 7.69 [10:02:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.51, 3.74 [10:04:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 4.07, 3.91 [10:08:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:10:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.68, 3.80 [10:12:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 4.39, 4.07 [10:12:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.19, 2.82 [10:14:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.80, 2.71 [10:16:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.72, 3.89 [10:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:26:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 2.64, 3.30 [10:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:36:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.67, 3.47 [10:38:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.37, 2.80 [10:38:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.30, 3.73 [10:40:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.43, 3.01 [10:40:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.99, 3.69 [10:41:19] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:42:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 4.98, 4.10 [10:44:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.96, 3.35 [10:44:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.50, 3.19, 2.94 [10:46:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.32, 3.18 [10:48:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.84, 3.89 [10:52:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:53:02] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:54:38]