[00:01:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:02:49] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:02:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.37, 9.25, 9.00 [00:03:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.75, 3.05 [00:04:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 3.78, 3.11 [00:04:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.51, 9.19, 9.02 [00:05:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 3.84, 3.10 [00:05:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.69, 8.42, 8.17 [00:05:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.41, 3.06 [00:06:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.17, 9.44, 9.12 [00:06:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.87, 8.79, 7.92 [00:07:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.47, 3.05 [00:07:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.87, 7.59, 7.91 [00:08:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 9.58, 9.21 [00:08:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.28, 8.58, 8.51 [00:09:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 3.17, 3.00 [00:09:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.90, 3.41 [00:10:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 3.82, 3.30 [00:10:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.22, 9.73, 8.46 [00:12:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.74, 8.39, 8.47 [00:12:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 9.78, 8.64 [00:13:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 4.05, 3.40 [00:13:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 8.51, 8.20 [00:14:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 3.30, 3.22 [00:15:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.68, 3.34 [00:15:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.39, 3.36 [00:17:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.48, 2.96, 3.12 [00:17:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.57, 8.36, 8.24 [00:18:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.95, 8.12, 8.27 [00:19:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.95, 3.58 [00:20:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.82, 8.24, 7.57 [00:21:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.98, 8.52 [00:21:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.65, 3.52 [00:22:17] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.97, 7.90, 7.54 [00:22:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 8.63, 8.50 [00:23:34] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:24:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.98, 8.26, 8.37 [00:25:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:28:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.79, 8.07, 8.45 [00:29:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.59, 3.48 [00:29:31] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [00:31:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.84, 3.23, 3.35 [00:32:49] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:36:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.55, 8.72, 8.65 [00:36:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.66, 7.68, 8.33 [00:38:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.42, 8.28, 8.49 [00:39:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 8.76, 8.31 [00:40:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 8.93, 8.69 [00:42:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.97, 8.65, 8.63 [00:43:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.18, 9.45, 8.72 [00:43:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 8.40, 7.76 [00:44:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.81, 3.42 [00:46:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.59, 3.39 [00:47:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.83, 8.58, 8.58 [00:47:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.01, 7.99, 7.77 [00:48:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.20, 3.27 [00:49:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.16, 7.85, 8.32 [00:51:34] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:55:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 8.23, 8.47 [00:56:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.59, 8.01, 8.46 [00:58:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 8.76, 8.53 [00:58:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:00:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.11, 8.21, 8.36 [01:01:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:04:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.94, 1.41 [01:06:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:08:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.46, 1.34 [01:11:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.11, 8.21, 7.52 [01:11:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.49, 9.42, 8.58 [01:11:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 8.87, 8.50 [01:12:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.17, 9.33, 8.65 [01:13:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 8.55, 8.42 [01:14:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.85, 1.60 [01:15:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.45, 8.01, 7.62 [01:16:11] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.50, 1.40, 1.46 [01:17:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.48, 10.11, 9.04 [01:17:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.27, 3.06 [01:18:08] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:18:08] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:19:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.80, 9.15, 8.82 [01:19:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 2.79, 2.90 [01:20:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.70, 9.13, 8.62 [01:24:11] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.69, 3.47, 2.31 [01:24:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.26, 8.32, 8.46 [01:26:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.84, 9.47, 8.90 [01:27:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.95, 3.85, 2.94 [01:28:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.70, 3.24 [01:28:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.03, 9.24, 8.89 [01:29:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 4.01, 3.00 [01:29:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 8.57, 8.17 [01:30:10] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.74, 1.87 [01:30:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 4.03, 3.42 [01:31:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.66, 2.99 [01:31:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.00, 8.17, 8.06 [01:32:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.54, 8.85, 7.92 [01:32:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.31, 9.22, 8.85 [01:32:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 8.67, 8.46 [01:33:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.84, 3.13 [01:34:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.31, 1.68 [01:34:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.28, 8.45, 7.88 [01:34:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.11, 8.63 [01:36:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.64, 9.22, 8.96 [01:36:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.28, 8.53, 8.49 [01:40:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 9.46, 9.13 [01:40:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.74, 10.01, 9.06 [01:41:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.84, 3.63 [01:42:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 9.35, 9.05 [01:42:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.64, 9.74, 9.07 [01:43:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.84, 3.57 [01:44:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.31, 9.47, 9.28 [01:44:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 8.82, 8.88 [01:44:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.18, 9.79, 9.16 [01:44:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.67, 4.41, 3.88 [01:45:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.07, 4.83, 3.95 [01:46:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.27, 9.17, 9.01 [01:48:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.74, 10.15, 9.55 [01:49:56] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.69, 1.34 [01:50:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.78, 9.34, 9.33 [01:51:56] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.54, 1.62, 1.36 [01:52:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:52:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.48, 10.00, 9.37 [01:55:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:56:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.25, 9.77, 9.55 [01:56:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.99, 9.12, 9.18 [01:58:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 10.13, 9.53 [01:58:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 8.87, 8.55 [01:59:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.79, 1.54 [02:00:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.90, 9.14, 9.33 [02:00:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.18, 8.30, 8.39 [02:00:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.57, 9.20 [02:01:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.67, 1.53 [02:02:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 9.74, 9.51 [02:02:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.56, 8.73, 8.92 [02:04:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.63, 10.18, 9.71 [02:05:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.36, 3.85 [02:06:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.21, 3.86 [02:06:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 9.59, 9.53 [02:06:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.05, 7.81, 8.49 [02:08:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.59, 9.52 [02:12:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.95, 9.34, 9.17 [02:12:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.38, 9.53, 9.55 [02:13:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.00, 2.57, 3.27 [02:14:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.20, 9.31, 9.18 [02:14:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.46, 9.88, 9.64 [02:16:39] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:16:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 9.53, 9.54 [02:17:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.03, 8.84, 8.50 [02:20:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.08, 10.66, 9.70 [02:21:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 9.16, 8.70 [02:22:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 4.06, 3.86 [02:24:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.42, 9.04, 9.28 [02:25:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.77, 8.39, 8.49 [02:25:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:28:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.83, 3.83 [02:28:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.68, 3.42 [02:28:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:30:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.19, 3.27 [02:32:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.50, 9.25 [02:34:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 9.59, 9.33 [02:36:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.89, 3.85 [02:36:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 8.75, 8.61 [02:37:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 8.45, 8.27 [02:38:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.42, 3.68 [02:39:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.56, 8.09, 8.17 [02:40:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.15, 9.39, 9.27 [02:42:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.78, 8.33, 8.50 [02:43:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.90, 9.66, 8.77 [02:44:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.44, 3.57 [02:44:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.36, 9.32 [02:45:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.89, 8.82, 8.56 [02:45:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.01, 3.39 [02:47:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [02:47:49] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:49:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.30, 8.15, 8.36 [02:49:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.57, 1.19 [02:50:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.35, 3.94 [02:51:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.71, 1.27, 1.12 [02:53:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:54:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.72, 7.42, 8.31 [02:55:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.68, 9.50, 8.85 [02:56:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.16, 9.05, 8.77 [02:58:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.89, 3.89 [02:58:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:59:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:59:34] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406804084777832 seconds [03:00:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.49, 3.74 [03:00:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 7.79, 8.29 [03:02:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.43, 3.96, 3.89 [03:03:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:04:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.60, 3.77 [03:08:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.99, 3.84 [03:08:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 8.75, 8.72 [03:10:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 8.54, 8.06 [03:10:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.79, 9.16, 8.88 [03:10:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.37, 8.85, 8.37 [03:12:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.85, 7.60, 7.78 [03:12:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.86, 3.84 [03:12:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.69, 9.44, 9.00 [03:12:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.05, 8.65, 8.36 [03:13:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:14:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 4.14, 3.94 [03:14:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.32, 9.38, 9.03 [03:14:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.45, 7.64, 8.04 [03:16:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.72, 3.81 [03:17:05] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [03:18:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:18:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.45, 9.11 [03:20:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.41, 3.97, 3.85 [03:20:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.93, 8.98, 8.99 [03:22:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 2.89, 2.74 [03:23:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.25, 4.05, 3.16 [03:28:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.74, 3.23 [03:28:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:29:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 8.50, 7.97 [03:30:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.38, 8.07, 8.47 [03:31:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 8.72, 8.11 [03:32:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.82, 3.34 [03:33:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 9.46, 8.46 [03:34:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.67, 3.35 [03:35:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:36:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.98, 3.39, 3.28 [03:37:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.35, 9.31, 8.65 [03:37:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.88, 8.37, 7.71 [03:38:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.50, 3.93 [03:39:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.61, 10.14, 9.03 [03:40:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.70, 3.94 [03:40:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.70, 9.05, 8.73 [03:40:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.35, 9.05, 8.38 [03:41:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.90, 9.58, 8.98 [03:41:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.26, 8.16, 7.22 [03:41:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.53, 8.14, 7.79 [03:42:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.70, 3.91 [03:42:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.58, 8.19, 8.46 [03:42:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.25, 9.18, 8.51 [03:44:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.64, 9.95, 9.25 [03:46:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.27, 8.28, 8.29 [03:47:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 8.38, 7.64 [03:48:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.46, 2.29, 3.23 [03:48:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.27, 8.62, 8.92 [03:50:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:52:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 9.50, 9.14 [03:54:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.75, 8.77, 8.92 [03:58:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.55, 9.15 [03:59:31] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [04:00:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 9.33, 9.11 [04:02:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:10:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 7.95, 8.49 [04:21:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:32:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.85, 8.57, 8.02 [04:34:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.44, 8.46, 8.07 [04:47:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:57:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:13:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.58, 2.89 [05:19:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.90, 3.28 [05:20:01] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.80, 1.43 [05:21:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:01] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.59, 1.67, 1.42 [05:23:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.21, 3.14 [05:26:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:00:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.92, 8.85, 7.67 [06:02:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 8.21, 7.58 [06:06:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.39, 0.94 [06:08:06] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.17, 0.91 [06:08:50] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/ef172831b919...07515cb55e0c [06:08:52] [02miraheze/ssl] 07Reception123 0307515cb - readd wikipedia.yogfront.ooo cert [06:17:00] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.51, 1.14 [06:19:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.48, 1.25 [06:20:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.41, 1.15, 1.09 [06:21:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.39, 1.25 [06:27:40] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/9b60cd834740...1f2ecbb94260 [06:27:42] [02miraheze/dns] 07Reception123 031f2ecbb - add burnout.wiki zone [06:29:06] Reception123, have you finished having to renew all the SSLs manually now, so they're all back to being auto-renewed by MirahezeSSLBot? [06:29:32] dmehus: oh yes a long time ago [06:29:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [06:29:34] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4040398597717285 seconds [06:29:41] Reception123, oh, that's good :) [06:44:01] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://github.com/miraheze/ssl/compare/07515cb55e0c...81b3d603bbbc [06:44:03] [02miraheze/ssl] 07Reception123 0317ce402 - add entw.ml cert [06:44:04] [02miraheze/ssl] 07Reception123 0381b3d60 - add burnout.wiki cert [06:47:41] !log DELETED and DROPPED torberawiki (reset request) [06:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:04] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.98, 1.71, 1.49 [06:52:31] !log DELETED and DROPPED eznirwiki (reset request) [06:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:04] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.12, 1.86, 1.57 [06:56:04] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.90, 1.86, 1.61 [06:56:36] !log RENAMED samanthapersosnwiki to samanthapersonswiki (T8898) [06:56:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:00:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.32, 1.60, 1.58 [07:00:53] [02mw-config] 07Reception123 closed pull request 03#4489: T8867:Update with missing WikiSEO params for GP&GD - 13https://github.com/miraheze/mw-config/pull/4489 [07:00:55] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3e4ad15b78f8...78cbc2ed58e8 [07:00:56] [02miraheze/mw-config] 07ugochimobi 0378cbc2e - T8867:Update with missing WikiSEO params for GP&GD (#4489) [07:00:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:02:05] miraheze/mw-config - Reception123 the build passed. [07:18:22] !log [@test101] starting deploy of {'config': True} to all [07:18:23] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [07:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:22:18] !log [@mwtask111] starting deploy of {'config': True} to all [07:22:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:22:29] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [07:22:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [07:30:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403279781341553 seconds [07:48:22] [02puppet] 07Universal-Omega opened pull request 03#2449: Varnish: Fix typo for unsetting authorization http header - 13https://github.com/miraheze/puppet/pull/2449 [07:56:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.72, 1.18 [07:57:46] [02puppet] 07Reception123 closed pull request 03#2449: Varnish: Fix typo for unsetting authorization http header - 13https://github.com/miraheze/puppet/pull/2449 [07:57:47] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7447bf95c0d0...d79f177324b6 [07:57:49] [02miraheze/puppet] 07Universal-Omega 03d79f177 - Varnish: Fix typo for unsetting authorization http header (#2449) [07:58:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.42, 1.13 [08:00:48] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [08:26:57] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [08:54:57] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:15:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:30:32] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [09:30:33] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406766891479492 seconds [09:39:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.20, 2.26 [09:41:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 2.93, 2.27 [10:00:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [10:13:58] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:14:04] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.95, 3.29, 1.80 [10:18:04] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.71, 1.83, 1.53 [10:20:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.59, 1.40, 1.41 [10:49:26] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.68, 1.39 [10:51:26] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.51, 1.36 [11:16:53] [02mw-config] 07ugochimobi opened pull request 03#4490: Update with missing default Kartographer params - 13https://github.com/miraheze/mw-config/pull/4490 [11:17:58] miraheze/mw-config - ugochimobi the build passed. [11:56:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.94, 4.09, 3.34 [11:58:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.49, 3.21 [11:59:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.05, 3.07 [12:00:08] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [12:00:08] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4030890464782715 seconds [12:14:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:23:18] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.44, 1.99, 1.42 [12:25:15] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.55, 1.83, 1.44 [12:29:10] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.50, 1.69, 1.47 [12:30:01] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [12:32:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.16, 4.42, 3.07 [12:32:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:33:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.22, 4.80, 3.32 [12:34:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.90, 2.87 [12:35:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.49, 8.52, 7.75 [12:36:33] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.83, 3.26, 2.76 [12:36:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.87, 3.20 [12:37:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.84, 3.23 [12:37:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:37:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.77, 8.04, 7.68 [12:38:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.96, 3.31 [12:39:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 3.28, 3.10 [12:40:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.83, 3.23, 3.12 [12:41:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 8.61, 8.02 [12:43:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.38, 8.45, 8.05 [12:43:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:43:52] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:47:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.23, 3.08 [12:49:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 8.95, 8.30 [12:49:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 2.93, 2.98 [12:51:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.15, 8.23, 8.12 [12:53:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:59:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 3.91, 3.33 [12:59:57] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [12:59:57] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405285835266113 seconds [13:00:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 8.42, 7.83 [13:01:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.39, 3.20 [13:02:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.39, 7.67, 7.61 [13:02:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:10:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:16:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.19, 4.20, 3.56 [13:18:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.78, 3.25, 3.29 [13:25:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.21, 9.01, 8.33 [13:34:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 8.97, 8.40 [13:40:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.06, 10.05, 8.97 [13:41:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 8.79, 8.02 [13:45:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.70, 8.03, 7.91 [13:46:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.23, 9.57, 9.17 [13:52:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:54:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.65, 7.68, 8.38 [14:05:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.93, 3.32 [14:06:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.13, 2.65 [14:07:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.85, 3.17, 3.12 [14:08:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 2.79, 2.58 [14:09:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:10:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.61, 9.41, 8.50 [14:11:11]