[00:02:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.34, 2.27, 1.44 [00:02:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.22, 2.68 [00:04:40] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 788 failures. Last run 3 minutes ago with 788 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:04:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.30, 8.75, 8.04 [00:06:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.68, 4.03, 3.47 [00:06:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.16, 8.97, 8.97 [00:06:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.34, 2.00, 1.53 [00:06:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.41, 2.91 [00:06:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.44, 8.14, 7.89 [00:08:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.89, 7.57, 6.95 [00:08:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.17, 2.76, 1.88 [00:08:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.85, 9.06, 9.02 [00:09:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.77, 3.17 [00:10:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.10, 7.42, 6.99 [00:10:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.75, 3.17 [00:11:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.93, 4.09, 3.36 [00:11:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.83, 9.58, 8.82 [00:12:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.41, 1.53, 1.58 [00:12:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.67, 3.22 [00:13:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.44, 3.49, 3.23 [00:13:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.62, 8.49, 8.52 [00:14:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.68, 7.65, 8.36 [00:14:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.88, 3.35 [00:15:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.38, 3.23 [00:18:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.78, 3.45 [00:19:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.53, 3.38 [00:22:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.59, 3.77 [00:22:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.33, 3.33 [00:22:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.92, 7.79, 8.35 [00:23:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.92, 2.90, 3.18 [00:28:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 2.92, 3.38 [00:32:51] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.91, 8.48, 8.21 [00:34:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.71, 8.07, 8.12 [00:36:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:37:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.41, 3.18 [00:39:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.14, 3.10 [00:46:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:51:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:56:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:58:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [00:58:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405253171920776 seconds [00:59:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:01:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.37, 9.14, 8.60 [01:03:47] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.42, 9.71, 8.85 [01:07:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 9.45, 8.99 [01:09:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.93, 10.07, 9.27 [01:09:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:11:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.86, 9.81, 9.26 [01:12:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.63, 8.75, 8.26 [01:12:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.87, 1.21 [01:14:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.52, 1.35, 1.10 [01:18:55] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:18:55] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:20:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 8.25, 8.27 [01:21:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.32, 2.03, 1.45 [01:23:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.70, 1.40 [01:24:35] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:51] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:55] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.43, 7.62, 8.37 [01:25:02] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:32] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.358 second response time [01:26:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:26:49] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [01:27:02] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [01:31:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:33:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.72, 8.90, 8.60 [01:34:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.86, 9.17, 8.53 [01:35:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 8.15, 8.36 [01:35:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.23, 8.62, 7.62 [01:36:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.65, 8.73, 7.88 [01:37:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.55, 8.52, 7.71 [01:38:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.53, 8.30, 7.83 [01:40:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.64, 8.99, 8.72 [01:41:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.30, 7.88, 7.63 [01:44:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.40, 7.82, 8.33 [01:46:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 8.12, 8.05 [01:50:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:52:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.11, 7.85, 7.99 [01:55:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:58:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:00:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.92, 8.04, 7.87 [02:02:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.89, 7.65, 7.76 [02:03:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 9.70, 8.79 [02:10:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.35, 9.51, 9.16 [02:11:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.11, 9.10, 8.26 [02:12:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.09, 9.74, 9.29 [02:13:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.77, 8.51, 8.15 [02:14:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.44, 9.21, 9.15 [02:15:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.85, 8.43, 8.17 [02:16:44] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.37, 10.12, 9.48 [02:17:40] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:19:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.21, 8.82, 8.38 [02:21:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.21, 9.18, 8.56 [02:22:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 9.99, 9.65 [02:23:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.34, 2.81 [02:23:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.31, 9.03, 8.59 [02:25:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.60, 8.38, 8.42 [02:26:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.46, 2.92 [02:27:31] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [02:28:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:28:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.10, 2.86 [02:43:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.79, 9.05, 8.49 [02:45:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.44, 9.34, 8.64 [02:48:58] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [02:48:59] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:49:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.31, 9.60, 8.93 [02:50:53] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:54:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.77, 7.95, 8.41 [02:59:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 7.98, 7.54 [03:00:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.35, 9.24, 8.82 [03:01:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.35, 9.75, 9.15 [03:01:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.97, 7.64, 7.47 [03:02:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.63, 9.49, 8.97 [03:04:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.53, 9.53, 7.90 [03:04:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.58, 9.76, 9.33 [03:06:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.75, 8.69, 7.78 [03:08:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.93, 8.19, 7.71 [03:10:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.52, 9.28, 9.18 [03:11:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 8.49, 7.99 [03:12:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 8.85, 8.15 [03:14:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 8.57, 8.00 [03:14:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 8.61, 8.05 [03:14:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 9.11, 8.31 [03:14:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.69, 9.42, 9.33 [03:15:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.42, 8.40, 8.10 [03:16:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.30, 8.09, 7.93 [03:18:21] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:18:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.11, 9.60, 9.35 [03:19:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.30, 7.92, 7.98 [03:20:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.31, 9.62, 8.92 [03:22:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.14, 9.80, 9.14 [03:24:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.24, 9.28, 9.04 [03:24:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.32, 9.49, 9.47 [03:29:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:30:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 8.41, 8.09 [03:30:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.18, 9.82, 9.30 [03:32:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.32, 9.18, 8.39 [03:32:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:34:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.73, 9.56, 9.07 [03:35:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 8.68, 8.14 [03:36:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 9.14, 8.96 [03:36:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 9.59, 8.80 [03:37:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.25, 7.86, 7.91 [03:38:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 9.81, 9.68 [03:40:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.03, 8.13, 8.43 [03:40:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.73, 10.73, 10.04 [03:40:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.13, 10.02, 9.27 [03:42:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.51, 7.93, 8.48 [03:42:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.26, 9.01, 9.49 [03:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 9.57, 9.21 [03:44:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.62, 10.05, 9.81 [03:46:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.36, 8.90, 9.41 [03:47:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:52:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.40, 2.88 [03:54:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.28, 2.89 [03:54:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.88, 9.28, 9.02 [03:56:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.33, 9.02, 8.94 [03:58:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.48, 9.78, 9.48 [03:58:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:00:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 9.82, 9.22 [04:02:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.39, 9.32, 9.12 [04:06:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.72, 3.13 [04:08:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.49, 3.11 [04:10:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.64, 10.03, 9.43 [04:12:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.18, 9.56, 9.36 [04:14:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.18, 3.31, 3.18 [04:17:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.50, 3.25 [04:18:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.38, 9.57, 9.34 [04:20:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.33, 8.97, 8.47 [04:20:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.95, 1.91, 1.27 [04:22:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.61, 1.24 [04:22:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.47, 9.25, 9.85 [04:22:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.90, 9.89, 9.53 [04:22:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 3.00, 3.15 [04:24:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.10, 9.82, 9.54 [04:26:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.89, 8.40, 8.39 [04:26:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.25, 9.21, 9.35 [04:30:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.56, 10.24, 9.93 [04:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:34:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.02, 8.52, 7.89 [04:34:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 7.86, 7.56 [04:36:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.83, 7.90, 7.75 [04:36:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:36:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.24, 7.35, 7.42 [04:37:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.76, 1.48 [04:38:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.52, 9.15, 9.65 [04:39:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.45, 1.28, 1.33 [04:40:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 8.90, 8.89 [04:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 5.91, 7.85, 8.52 [04:44:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.68, 6.81, 8.48 [04:44:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.30, 7.54, 8.31 [04:48:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.16, 2.87 [04:50:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.53, 2.67 [04:51:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:02:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.09, 8.53, 8.37 [05:03:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 3.43, 2.91 [05:05:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.40, 3.20, 2.88 [05:05:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.65, 8.39, 7.60 [05:05:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:06:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.47, 8.11, 8.23 [05:07:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.62, 7.94, 7.54 [05:14:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.26, 8.95, 8.61 [05:15:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:16:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.47, 8.40, 8.45 [05:20:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.26, 9.22, 8.75 [05:22:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.37, 8.51, 8.55 [05:22:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:24:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.14, 7.85, 8.29 [05:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:12:33] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.90, 1.47, 0.85 [06:14:33] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.57, 1.13, 0.80 [06:16:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.00, 1.76, 1.21 [06:22:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.80, 2.09, 1.51 [06:26:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.70, 1.98, 1.62 [06:32:17] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') Fil [06:32:18] lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.402985095977783 seconds [06:37:23] !log ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title), ADD FULLTEXT INDEX si_text (si_text); (hbrwiki) [06:37:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.00, 7.78, 7.34 [06:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:42:23] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.37, 1.65, 1.66 [06:45:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.00, 7.97, 7.56 [06:45:47] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=massivestardatabasewiki --report 1 /home/reception/Wikipedia-20220306063930.xml (START) [06:45:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:46:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:47:21] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=massivestardatabasewiki --report 1 /home/reception/Wikipedia-20220306063930.xml (END - exit=0) [06:47:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:23] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/dc32d86fba6a...2b3cfd5808c8 [06:58:24] [02miraheze/MirahezeMagic] 07Reception123 032b3cfd5 - fix link [07:00:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.07, 2.48 [07:02:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 2.76, 2.44 [07:10:16] miraheze/MirahezeMagic - Reception123 the build passed. [07:32:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.19, 2.78 [07:33:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.28, 2.66 [07:34:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.65, 2.63 [07:35:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.20, 2.68 [07:37:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.44, 2.84 [07:37:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.47, 3.62, 2.77 [07:38:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.56, 3.00 [07:39:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 3.33, 2.90 [07:39:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.24, 2.74 [07:40:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.07, 2.89 [07:44:09] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.07, 6.37, 5.44 [07:46:09] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.03, 6.32, 5.54 [07:53:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 3.80, 2.92 [07:54:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.61, 3.02 [07:55:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.74, 3.09, 2.78 [07:56:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.50, 2.82, 2.80 [07:57:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:58:51] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:02:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:52:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [08:52:26] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:52:59] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [08:52:59] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404659271240234 seconds [09:21:48] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:30:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.27, 0.84 [09:32:33] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.06, 0.81 [09:50:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.50, 1.25 [09:51:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:51:10] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:52:21] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [09:54:31] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.03, 1.33, 1.25 [09:58:46] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.leapit.no' expires in 15 day(s) (Tue 22 Mar 2022 09:54:24 GMT +0000). [10:04:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1957a04eeacc...5111dd994462 [10:04:59] [02miraheze/ssl] 07MirahezeSSLBot 035111dd9 - Bot: Update SSL cert for wiki.leapit.no [10:20:32] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:22:02] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [10:22:02] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406431198120117 seconds [10:24:31] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hoseki.wiki' expires in 15 day(s) (Tue 22 Mar 2022 10:02:02 GMT +0000). [10:28:32] RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Sat 04 Jun 2022 09:04:51 GMT +0000. [10:29:34] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') Fil [10:29:34] lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407342195510864 seconds [10:31:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5111dd994462...d4cea6d95b59 [10:31:26] [02miraheze/ssl] 07MirahezeSSLBot 03d4cea6d - Bot: Update SSL cert for hoseki.wiki [10:49:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.04, 0.63 [10:50:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.22, 2.29 [10:51:35] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 0.96, 0.65 [10:54:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.37, 2.63 [10:58:22] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:21:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [11:22:31] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sat 04 Jun 2022 09:31:19 GMT +0000. [11:51:05] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:51:05] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40658164024353 seconds [12:06:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:11:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:20:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.38, 1.77, 1.28 [12:34:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.64, 1.93, 1.74 [12:37:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:39:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.87, 8.87, 7.41 [12:41:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.84, 8.33, 7.38 [12:43:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 9.24, 8.26 [12:43:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 8.97, 8.04 [12:44:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.89, 1.79 [12:44:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 8.12, 7.07 [12:45:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.22, 9.86, 8.48 [12:46:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.58, 1.78, 1.76 [12:46:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 7.71, 7.05 [12:47:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.53, 9.55, 8.52 [12:47:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:49:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.04, 9.49, 8.61 [12:50:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.31, 1.86, 1.78 [12:50:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [12:50:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.39, 8.50, 8.30 [12:51:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.15, 8.87, 8.48 [12:51:41] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:55:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.15, 8.32, 8.37 [13:00:23] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.51, 1.83, 1.87 [13:01:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.41, 9.06, 8.67 [13:04:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.79, 9.13, 8.72 [13:04:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.63, 8.29, 7.92 [13:06:41] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:06:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.42, 8.40, 8.50 [13:08:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.06, 7.58, 7.77 [13:13:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:16:23] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.97, 1.42, 1.66 [13:16:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.35, 2.01, 1.38 [13:17:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 8.30, 7.65 [13:18:33] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.79, 1.55, 1.28 [13:19:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.57, 7.96, 7.60 [13:20:08] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:20:08] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404741287231445 seconds [13:20:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.71, 1.35 [13:20:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.03, 8.86, 8.30 [13:22:31] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.56, 1.34 [13:24:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.32, 8.97, 8.50 [13:26:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.19, 8.35, 8.32 [13:28:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.22, 2.77 [13:34:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 2.69, 2.62 [13:36:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.89, 8.27, 8.11 [13:38:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:40:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 8.76, 8.29 [13:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.74, 8.90, 8.41 [13:43:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:44:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.82, 8.46, 8.31 [13:47:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.41, 3.05 [13:49:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.54, 2.77, 2.85 [13:49:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:49:45] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [13:52:41] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:52:41] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:53:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.63, 8.79, 8.52 [13:57:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.07, 3.54, 3.00 [13:57:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 9.31, 8.74 [13:59:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 9.01, 8.71 [14:00:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.48, 10.70, 9.16 [14:01:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 3.10, 2.98 [14:03:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.34, 8.07, 8.43 [14:04:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.76, 9.81, 9.15 [14:11:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.92, 8.98, 8.66 [14:11:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.28, 7.96, 7.26 [14:13:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.21, 7.89, 7.31 [14:16:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.04, 2.78 [14:16:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.21, 7.99, 8.44 [14:18:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 2.72, 2.70 [14:20:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 8.51, 8.55 [14:22:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.87, 8.37, 8.50 [14:24:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:25:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.83, 8.46, 7.61 [14:26:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.46, 9.78, 9.03 [14:27:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.34, 7.61, 7.40 [14:31:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.47, 2.99 [14:32:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.18, 2.91 [14:33:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.23, 2.97 [14:34:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.09, 3.08, 2.90 [14:34:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.51, 9.82, 9.40 [14:35:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:36:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.06, 8.16, 8.49 [14:37:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.59, 3.19 [14:42:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.74, 9.02, 8.75 [14:44:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.90, 3.45 [14:46:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.18, 4.35, 3.67 [14:46:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.09, 9.22, 9.06 [14:48:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.78, 8.87, 8.95 [14:49:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.81, 3.62 [14:50:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 8.66, 7.91 [14:50:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.69, 3.73 [14:50:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.78, 9.58, 9.02 [14:51:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:52:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 7.92, 7.20 [14:53:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.06, 4.08, 3.37 [14:53:47]