[00:01:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.09, 3.69, 2.88 [00:02:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.98, 3.43, 2.90 [00:03:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.96, 1.35 [00:03:22] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 452 failures. Last run 2 minutes ago with 452 failures. Failed resources (up to 3 shown): File[opendatascot.org_private],File[savagepedia.wiki],File[savagepedia.wiki_private],File[wiki.anglish.info] [00:03:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.55, 2.92 [00:05:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 1.65, 1.31 [00:05:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.12, 3.35, 2.92 [00:07:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 8.68, 7.78 [00:11:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 8.84, 7.99 [00:11:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.97, 3.33 [00:11:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.86, 8.54, 7.75 [00:11:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.61, 8.15, 7.77 [00:11:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.79, 10.51, 9.23 [00:12:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.28, 3.15 [00:13:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 8.34, 7.92 [00:13:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.18, 8.20, 7.73 [00:15:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.51, 9.56, 9.17 [00:16:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.38, 4.13, 3.52 [00:17:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.31, 10.01, 9.38 [00:19:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.82, 1.29 [00:19:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.83, 9.25, 9.17 [00:21:07] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.31, 1.55, 1.25 [00:21:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.97, 3.72 [00:22:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.89, 3.65 [00:23:57] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.04, 7.34, 6.70 [00:24:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:27:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 2.90, 3.31 [00:28:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.45, 3.47 [00:29:57] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.10, 7.59, 7.06 [00:30:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.85, 3.25 [00:31:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:31:41] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [00:33:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.26, 7.89, 8.48 [00:34:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.44, 4.63, 3.82 [00:34:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.59, 3.03 [00:35:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.58, 3.42 [00:35:57] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.17, 7.14, 6.98 [00:36:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.57, 3.10 [00:37:57] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 6.87, 6.89 [00:38:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.03, 3.15, 2.99 [00:38:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.19, 3.32 [00:39:57] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.37, 6.48, 6.76 [00:40:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.69, 3.71 [00:40:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.76, 9.24, 8.29 [00:41:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.51, 9.02, 9.73 [00:43:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.21, 8.62, 8.42 [00:44:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.92, 3.37 [00:45:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.66, 9.57, 9.79 [00:46:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.75, 8.48, 8.28 [00:50:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 8.75, 8.45 [00:51:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.03, 9.49, 9.86 [00:53:03] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.85, 1.68, 1.47 [00:54:17] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [00:54:17] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:55:00] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.35, 1.58, 1.46 [00:55:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.11, 8.26, 8.48 [00:55:39] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [00:58:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.18, 8.48, 8.48 [00:58:55] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.08, 1.89, 1.61 [01:00:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.31, 3.13 [01:04:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 3.00, 3.04 [01:04:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.23, 1.77, 1.68 [01:05:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.29, 7.37, 8.48 [01:06:40] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.31, 1.62, 1.63 [01:10:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.84, 2.05, 1.41 [01:11:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:12:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.74, 1.37 [01:14:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.37, 1.28 [01:19:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.57, 3.15 [01:21:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.28, 3.10 [01:22:13] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.15, 1.65, 1.51 [01:23:04] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:24:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.48, 1.65, 1.53 [01:28:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.14, 9.99, 8.63 [01:38:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.33, 9.73, 8.70 [01:42:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.72, 9.39, 8.38 [01:43:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 8.45, 7.33 [01:43:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:45:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.04, 7.87, 7.25 [01:45:44] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.52, 8.11, 8.41 [01:46:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.53, 8.44, 8.24 [01:51:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:55:30] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [01:55:30] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407233953475952 seconds [01:58:11] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.63, 9.17, 8.63 [02:00:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.37, 8.81, 8.56 [02:02:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.59, 9.79, 8.95 [02:05:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.97, 9.26, 8.96 [02:11:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:13:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.00, 9.41, 8.27 [02:13:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.03, 9.72, 9.17 [02:15:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.60, 9.44, 8.43 [02:15:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.81, 9.31, 9.06 [02:15:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:16:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 8.83, 7.88 [02:17:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.25, 8.31, 8.14 [02:20:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.27, 8.34, 7.91 [02:20:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:28:40] RECOVERY - wiki.meower.org - reverse DNS on sslhost is OK: SSL OK - wiki.meower.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:29:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.05, 8.91 [02:30:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.86, 1.72, 1.47 [02:31:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.95, 9.36, 9.04 [02:31:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:32:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.24, 1.57, 1.44 [02:35:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.92, 8.98, 8.43 [02:36:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.86, 8.80, 7.98 [02:36:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.11, 2.74 [02:38:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.30, 8.93, 7.98 [02:38:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.04, 2.76 [02:40:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.94, 9.02, 8.13 [02:40:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.39, 7.86, 7.81 [02:41:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.87, 8.01, 8.22 [02:41:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.44, 7.65, 8.34 [02:41:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:44:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.97, 7.79, 7.88 [02:44:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.67, 9.47, 8.79 [02:49:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [02:58:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.03, 3.25 [03:01:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:02:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.77, 3.34 [03:03:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.59, 9.62, 8.58 [03:05:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.11, 9.52, 8.68 [03:10:27] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.03, 3.27, 3.27 [03:11:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.82, 9.50, 9.01 [03:13:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.43, 9.56, 9.10 [03:14:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.85, 8.85, 8.32 [03:16:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:18:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.27, 7.99, 8.15 [03:18:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 8.60, 8.01 [03:19:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.04, 9.63, 8.51 [03:20:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.23, 7.64, 7.73 [03:20:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:21:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.87, 9.77, 8.72 [03:23:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.69, 9.33, 9.09 [03:23:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.79, 9.24, 8.85 [03:25:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.49, 8.74, 8.92 [03:25:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [03:25:21] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4068992137908936 seconds [03:27:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.84, 8.45, 8.48 [03:27:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.42, 8.95, 8.83 [03:29:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.02, 8.96 [03:29:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.14, 10.48, 9.40 [03:31:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.79, 1.33 [03:31:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 8.97, 8.96 [03:33:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.43, 1.25 [03:33:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.87, 9.88, 9.45 [03:39:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.90, 1.94, 1.64 [03:39:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.79, 9.28, 8.95 [03:39:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.32, 10.16, 9.65 [03:41:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.88, 9.69, 9.55 [03:43:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.67, 1.59 [03:45:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.99, 9.43, 9.21 [03:49:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.26, 8.72, 9.05 [03:51:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.51, 9.46, 9.18 [03:51:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.89, 9.26, 9.22 [03:53:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.19, 8.84, 8.98 [03:55:02] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.26, 2.07, 1.77 [03:55:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.41, 9.16 [03:57:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.70, 1.67 [03:57:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.87, 9.41, 9.22 [04:01:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.84, 9.43, 9.26 [04:05:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.37, 8.46, 8.96 [04:06:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.49, 8.81, 8.26 [04:10:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.19, 7.95, 8.04 [04:10:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:11:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.31, 7.69, 8.35 [04:11:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.42, 9.37, 9.09 [04:14:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.10, 8.52, 8.30 [04:15:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.90, 9.09, 8.79 [04:15:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.34, 9.72, 9.31 [04:17:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.12, 10.01, 9.45 [04:18:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.45, 8.35, 8.29 [04:19:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.59, 1.81, 1.38 [04:19:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.57, 8.63, 9.01 [04:21:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.74, 1.41 [04:23:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.13, 1.52, 1.36 [04:28:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 9.03, 8.53 [04:29:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.96, 8.00, 8.43 [04:30:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.37, 2.79 [04:31:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 9.48, 9.19 [04:32:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 8.14, 8.28 [04:32:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 2.79, 2.65 [04:33:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.52, 9.40, 9.20 [04:35:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.32, 9.16, 8.79 [04:39:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.78, 7.84, 8.37 [04:43:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.06, 8.63, 8.61 [04:47:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:49:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.48, 7.79, 8.48 [05:01:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 8.96, 8.79 [05:03:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.57, 8.07, 8.49 [05:03:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.94, 8.92, 8.37 [05:11:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.58, 8.82, 8.62 [05:13:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.44, 8.12, 8.38 [05:17:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:25:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.47, 7.97, 8.35 [05:41:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:46:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.54, 1.32 [05:51:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.37, 1.28 [05:51:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:06:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:10:09] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.10, 1.61, 1.29 [06:12:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.76, 1.68, 1.36 [06:14:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:19:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/e0f605c9eac6...252780cb344d [06:19:51] [02miraheze/ssl] 07Reception123 03252780c - add 3.141592653589793238462643383279502884197169399375105820974944592.ml cert [06:20:24] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/0a69d2f4291e...a887f39a45f9 [06:20:25] [02miraheze/dns] 07Reception123 03a887f39 - add zetapedia.com zone [06:20:54] [02dns] 07Reception123 commented on pull request 03#249: Remove Ovlon leftovers - 13https://github.com/miraheze/dns/pull/249#issuecomment-1059704427 [06:22:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.42, 1.97, 1.64 [06:22:37] [02puppet] 07Reception123 commented on pull request 03#2417: Tweak wording - 13https://github.com/miraheze/puppet/pull/2417#issuecomment-1059704628 [06:24:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.80, 1.97, 1.68 [06:25:30] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hbrwiki --username-prefix=wikia:heavenburnsred --no-updates --report 1 /home/reception/heavenburnsred_pages_full.xml (START) [06:25:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.23, 2.07, 1.76 [06:28:50] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=tuscriaturiaswiki (END - exit=65280) [06:29:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[3.141592653589793238462643383279502884197169399375105820974944592.ml],File[3.141592653589793238462643383279502884197169399375105820974944592.ml_private] [06:35:11] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hbrwiki --username-prefix=wikia:heavenburnsred --no-updates --report 1 /home/reception/heavenburnsred_pages_full.xml (END - exit=0) [06:35:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:35:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/252780cb344d...b3e2c235be13 [06:35:38] [02miraheze/ssl] 07Reception123 03b3e2c23 - attempt fix [06:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.74, 8.68, 8.17 [06:35:47] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hbrwiki (START) [06:35:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:02] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hbrwiki (END - exit=256) [06:36:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:24] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[3.141592653589793238462643383279502884197169399375105820974944592.ml],File[3.141592653589793238462643383279502884197169399375105820974944592.ml_private] [06:37:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.35, 8.27, 8.09 [06:37:58] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[3.141592653589793238462643383279502884197169399375105820974944592.ml],File[3.141592653589793238462643383279502884197169399375105820974944592.ml_private] [06:38:09] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=tuscriaturaswiki (END - exit=256) [06:38:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:19] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.18, 1.61, 1.21 [06:39:58] !log DROPPED searchindex on tuscriaturaswiki and replaced with backup after failed rebuildtextindex.php [06:40:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.58, 1.95, 1.91 [06:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:18] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.43, 1.19 [06:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:42:09] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 1.93, 2.01, 1.94 [06:44:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.80, 1.94, 1.93 [06:49:18] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.51, 1.30 [06:50:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.13, 1.77, 1.84 [06:50:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:51:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.41, 1.28 [06:56:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.51, 2.00, 1.97 [07:02:11] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.41, 1.43, 1.69 [07:02:49] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:04:08] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:05:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:29] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:13:50] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.79, 1.65, 1.64 [07:15:29] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.42, 8.32, 6.14 [07:15:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.59, 1.67, 1.65 [07:17:28] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.12, 7.71, 6.16 [07:19:27] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.50, 7.87, 6.39 [07:19:39] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.82, 1.80, 1.72 [07:21:35] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.39, 2.02, 1.81 [07:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:24:56] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:25:27] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.67, 1.98, 1.86 [07:31:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.35, 2.68, 1.82 [07:33:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.25, 1.83, 1.61 [07:33:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:35:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.63, 1.45, 1.50 [07:36:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:39:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.16, 1.79, 1.79 [07:39:16] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.50, 7.12, 7.97 [07:41:15] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.73, 8.63, 8.44 [07:41:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:45:13] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.68, 7.31, 7.96 [07:47:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.67, 1.63, 1.82 [07:47:12] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 13.14, 9.85, 8.81 [07:49:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.50, 1.28, 1.67 [07:50:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.76, 1.42 [07:52:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.54, 1.37 [07:57:06] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.50, 6.51, 7.68 [07:59:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 7.62, 7.00 [08:01:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.33, 8.71, 7.46 [08:02:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.45, 7.93, 7.14 [08:03:03] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 4.41, 5.11, 6.67 [08:03:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.22, 8.16, 7.41 [08:04:07] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.57, 7.67, 7.14 [08:06:59] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 11.22, 9.96, 8.31 [08:08:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 2.97, 2.48 [08:10:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.00, 2.56 [08:14:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.36, 2.81 [08:16:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.04, 2.77 [08:28:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:30:47] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.59, 6.60, 7.94 [08:32:46] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 15.10, 12.30, 10.00 [08:33:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.47, 1.97, 1.29 [08:37:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.51, 1.25 [08:42:40] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 6.61, 6.02, 7.77 [08:44:39] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 11.99, 8.06, 8.29 [08:48:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:54:18] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [08:54:33] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 5.48, 6.82, 7.86 [08:58:31] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 29.79, 13.96, 10.04 [09:06:27] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.24, 6.48, 7.86 [09:10:25] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.38, 3.64, 6.37 [09:12:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.96, 7.05, 5.65 [09:12:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:14:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.97, 6.57, 5.63 [09:22:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:46] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:23:46] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:40:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:45:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:13] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [09:50:13] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406340837478638 seconds [10:07:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:12:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:13:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.86, 8.43, 7.66 [10:15:51] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 8.29, 7.70 [10:28:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 2.99, 2.60 [10:30:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 2.80, 2.58 [10:44:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:48:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [10:54:12] [02dns] 07JohnFLewis closed pull request 03#249: Remove Ovlon leftovers - 13https://github.com/miraheze/dns/pull/249 [10:54:13] [02miraheze/dns] 07JohnFLewis pushed 033 commits to 03master [+0/-4/±0] 13https://github.com/miraheze/dns/compare/a887f39a45f9...81bc525b0a31 [10:54:15] [02miraheze/dns] 07Universal-Omega 03d415846 - Delete mh [10:54:16] [02miraheze/dns] 07Universal-Omega 031d713b5 - Delete 10.in-addr.arpa [10:54:18] [02miraheze/dns] 07JohnFLewis 0381bc525 - Merge pull request #249 from Universal-Omega/patch-11 [10:54:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:17:25] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:17:25] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405836582183838 seconds [11:19:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:21:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:24:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:31:52] [discord] Can someone clear my echo notifications on my account? [11:31:53] [discord] I have some alerts that refused to clear [11:31:55] [discord] RhinosF1 [11:43:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:46:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [11:48:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:51:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:56:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:08:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:11:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.53, 1.16 [12:13:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:15:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.73, 1.89, 1.38 [12:17:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.89, 1.87, 1.43 [12:20:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.65, 8.04, 6.83 [12:21:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.68, 2.11, 1.62 [12:22:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.94, 7.57, 6.81 [12:24:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:26:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.11, 9.22, 8.01 [12:28:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.12, 9.00, 8.10 [12:31:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.67, 1.90, 1.83 [12:32:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 8.83, 8.17 [12:36:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.83, 7.15, 5.51 [12:36:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.00, 8.50, 8.22 [12:37:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.02, 1.82, 1.80 [12:39:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:40:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.72, 8.50, 7.73 [12:42:18] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.88, 6.68, 5.87 [12:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:43:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.71, 1.85, 1.83 [12:44:06] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.85, 8.31, 7.85 [12:44:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.25, 8.66, 7.63 [12:44:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.39, 8.73, 8.41 [12:46:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.56, 7.33, 6.32 [12:46:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.57, 7.96, 7.51 [12:47:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.45, 2.03, 1.90 [12:48:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.76, 8.25, 6.79 [12:49:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.80, 1.90, 1.86 [12:50:01] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:50:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.29, 8.36, 8.46 [12:51:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.62, 2.13, 1.95 [12:52:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.76, 7.58, 6.91 [12:56:18] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.33, 6.32, 6.57 [13:03:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.81, 4.22, 2.42 [13:04:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 8.82, 8.50 [13:10:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.74, 2.77 [13:10:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.40, 8.12, 8.34 [13:11:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.16, 9.63, 8.36 [13:12:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.47, 2.79 [13:12:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:13:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.76, 1.87, 2.00 [13:13:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.97, 9.62, 8.51 [13:13:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:13:34] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405315399169922 seconds [13:14:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 2.95, 2.68 [13:14:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.01, 8.72, 8.59 [13:15:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 8.39, 7.61 [13:17:08]