[00:01:37] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1799 MB (19% inode=80%); [00:02:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.48, 4.37, 3.94 [00:03:11] RECOVERY - cp31 Disk Space on cp31 is OK: DISK OK - free space: / 5770 MB (14% inode=96%); [00:03:35] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 504 failures. Last run 2 minutes ago with 504 failures. Failed resources (up to 3 shown): File[index.pdcommunity.ir],File[index.pdcommunity.ir_private],File[spcodex.wiki],File[spcodex.wiki_private] [00:03:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.60, 3.18 [00:03:53] RECOVERY - cp30 Disk Space on cp30 is OK: DISK OK - free space: / 6054 MB (15% inode=96%); [00:05:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.95, 9.50, 8.62 [00:06:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.94, 3.43 [00:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:07:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.99, 3.49 [00:07:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 9.44, 8.70 [00:08:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.95, 3.93 [00:08:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 3.31, 3.26 [00:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.92, 7.88, 8.26 [00:13:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 8.37, 7.99 [00:14:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.92, 3.90 [00:15:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 8.92, 8.23 [00:15:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.39, 3.80 [00:15:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 8.41, 8.38 [00:17:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 8.76, 8.24 [00:17:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.89, 3.70 [00:17:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.25, 7.65, 8.09 [00:18:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.95, 3.93 [00:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:19:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.64, 8.91, 7.99 [00:21:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.52, 8.39, 7.92 [00:21:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.82, 4.21, 3.83 [00:22:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.58, 6.20, 5.11 [00:22:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:23:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.61, 3.65 [00:24:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.35, 6.32, 5.30 [00:25:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.86, 8.94, 8.56 [00:25:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.72, 9.27, 8.33 [00:26:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 2.77, 3.37 [00:27:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.69, 9.67, 8.88 [00:29:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 9.12, 8.77 [00:29:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.95, 8.94, 8.49 [00:31:33] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:31:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.93, 8.21, 8.27 [00:31:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.79, 3.66 [00:32:12] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.28, 2.62, 1.59 [00:32:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 8.13, 8.37 [00:33:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.11, 3.29, 3.50 [00:34:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.99, 1.98, 1.48 [00:35:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.07, 8.08, 8.48 [00:35:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.10, 3.40 [00:38:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.47, 1.39 [00:39:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.46, 3.54 [00:40:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 4.06, 3.56 [00:41:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.74, 8.65, 8.27 [00:41:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.52, 2.85, 3.31 [00:42:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 3.29, 3.33 [00:42:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.29, 8.85, 8.41 [00:43:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.54, 8.61, 8.44 [00:45:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.96, 9.52, 8.80 [00:47:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.88, 8.87, 8.65 [00:48:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.64, 8.26, 8.35 [00:48:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 8.80, 8.13 [00:49:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.76, 9.63, 8.95 [00:49:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.43, 8.73 [00:50:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:51:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 9.30, 8.78 [00:54:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.87, 9.38, 8.56 [00:55:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.37, 9.42, 9.10 [00:55:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.04, 8.25, 8.48 [00:56:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.50, 8.61, 8.37 [00:58:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.31, 8.34, 8.29 [01:00:56] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.57, 1.15 [01:01:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 9.30, 8.83 [01:02:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.06, 1.31, 1.10 [01:03:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.59, 7.47, 8.28 [01:03:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.26, 8.26, 8.51 [01:05:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 7.98, 8.38 [01:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:10:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.62, 9.14, 8.75 [01:11:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 8.41, 8.39 [01:12:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.83, 9.41, 8.57 [01:13:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.06, 8.09, 8.29 [01:13:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.05, 8.87, 8.76 [01:14:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.46, 3.07 [01:14:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 9.74, 8.81 [01:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:15:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.12, 8.02, 8.47 [01:16:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.49, 3.11 [01:20:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.61, 3.24 [01:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:20:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.02, 8.24, 8.45 [01:21:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.47, 8.91, 8.66 [01:22:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 3.10, 3.10 [01:23:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.94, 8.21, 8.44 [01:29:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [01:29:33] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.68, 10.71, 9.04 [01:33:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.04, 10.82, 9.25 [01:34:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 8.75, 8.43 [01:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.05, 9.05, 8.61 [01:37:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 8.59, 8.13 [01:39:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 9.24, 8.43 [01:39:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 9.98, 9.31 [01:40:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.33, 2.94 [01:41:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.07, 8.69, 8.32 [01:44:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.87, 3.31 [01:44:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.75, 8.28, 8.45 [01:45:10] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.18, 8.04, 8.17 [01:45:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 9.67, 9.55 [01:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.93, 3.22, 3.16 [01:49:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.71, 9.69, 8.78 [01:50:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.10, 2.74 [01:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:50:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.01, 9.78, 9.02 [01:51:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.93, 7.65, 8.43 [01:52:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.26, 4.00, 3.10 [01:52:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 9.51, 8.99 [01:53:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.70, 9.79, 9.09 [01:54:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.48, 3.26 [01:54:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.46, 3.00 [01:56:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.09, 3.15 [01:56:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.10, 2.93 [01:58:35] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:00:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 8.57, 8.42 [02:01:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.60, 9.11, 7.99 [02:02:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.74, 3.40 [02:02:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 9.77, 9.19 [02:04:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.28, 3.26 [02:05:22] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.95, 8.29, 7.92 [02:06:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 9.69, 9.35 [02:08:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.79, 3.50 [02:08:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.07, 8.15, 8.47 [02:08:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 9.93, 9.47 [02:10:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.34, 3.06 [02:11:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.19, 9.27, 9.02 [02:12:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.98, 3.09, 3.29 [02:12:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.17, 3.03 [02:12:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 9.03, 8.78 [02:12:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.29, 8.91 [02:13:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.42, 9.17, 9.03 [02:14:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 9.31, 9.37 [02:14:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.41, 9.53, 8.99 [02:15:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.54, 9.60, 9.20 [02:16:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.15, 9.19, 8.92 [02:16:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 9.66, 9.29 [02:18:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.46, 10.27, 9.74 [02:20:27] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.64, 1.33 [02:20:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.38, 9.51, 9.06 [02:20:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 9.53, 9.36 [02:22:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.63, 1.63, 1.36 [02:24:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.98, 9.79, 9.30 [02:26:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.49, 10.09, 9.45 [02:30:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.62, 9.79, 9.47 [02:31:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.79, 8.97, 8.07 [02:32:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.63, 9.31, 9.34 [02:32:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.40, 8.50, 7.99 [02:34:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 9.52, 9.77 [02:34:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.23, 8.07, 8.47 [02:34:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.41, 9.60, 9.43 [02:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 9.95, 9.88 [02:36:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.00, 9.30, 9.34 [02:40:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.76, 9.16, 8.79 [02:42:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.04, 8.02, 8.41 [02:43:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.27, 5.88, 5.11 [02:45:34] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.14, 5.27, 4.99 [02:46:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 8.58, 8.87 [02:46:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.42, 7.49, 8.37 [02:48:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.08, 8.67 [02:50:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 8.72, 8.57 [02:50:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.42, 8.30, 8.52 [02:52:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.49, 9.27, 9.01 [02:52:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.72, 8.33, 8.45 [02:54:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 9.28, 9.04 [02:56:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 8.84, 8.66 [02:57:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.19, 9.86, 9.74 [03:00:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.86, 7.77, 8.27 [03:00:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.81, 8.00, 8.39 [03:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.32, 8.59, 8.77 [03:06:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 9.02, 8.93 [03:09:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.71, 7.45, 8.38 [03:12:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.25, 7.15, 8.28 [03:15:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.62, 1.32 [03:21:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.55, 1.33 [03:21:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.34, 8.73, 8.35 [03:22:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 8.52, 8.24 [03:23:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.96, 8.40, 8.27 [03:23:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 8.23, 8.24 [03:30:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.43, 8.60, 8.44 [03:34:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.08, 8.50, 8.47 [03:35:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.19, 8.56, 8.11 [03:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.25, 8.95, 8.66 [03:37:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.65, 7.76, 7.87 [03:38:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.37, 8.46, 8.49 [03:39:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.89, 9.17, 8.59 [03:41:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 8.68, 8.24 [03:43:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.03, 9.04, 8.71 [03:45:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.28, 8.08, 8.40 [03:46:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.50, 8.56, 8.37 [03:48:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.94, 9.14, 8.59 [03:49:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.10, 9.02, 8.70 [03:50:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.84, 8.52, 8.30 [03:50:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 8.57, 8.55 [03:51:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.21, 8.10, 8.26 [03:51:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 9.44, 8.85 [03:52:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.71, 8.07, 8.15 [03:54:48] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Thu 31 Mar 2022 03:52:46 GMT +0000). [03:55:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.39, 9.92, 9.12 [03:56:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.50, 8.12, 8.37 [03:57:00] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [03:57:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.74, 9.63, 9.12 [03:59:32] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [03:59:32] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:00:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 8.84, 8.51 [04:02:29] PROBLEM - idleon.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'idleon.info' expires in 15 day(s) (Thu 31 Mar 2022 03:58:21 GMT +0000). [04:02:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9611fb394e50...30e7acb970b5 [04:02:45] [02miraheze/ssl] 07MirahezeSSLBot 0330e7acb - Bot: Update SSL cert for wiki.macc.nyc [04:04:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.53, 9.22, 8.70 [04:06:32] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.gq' expires in 15 day(s) (Thu 31 Mar 2022 03:56:46 GMT +0000). [04:06:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.55, 8.48, 8.49 [04:08:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/30e7acb970b5...34f609750e27 [04:08:23] [02miraheze/ssl] 07MirahezeSSLBot 0334f6097 - Bot: Update SSL cert for idleon.info [04:09:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.04, 7.61, 8.29 [04:12:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/34f609750e27...099f13ff5c17 [04:12:17] [02miraheze/ssl] 07MirahezeSSLBot 03099f13f - Bot: Update SSL cert for miraheze.gq [04:12:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.61, 8.62, 8.33 [04:15:40] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Thu 31 Mar 2022 03:50:41 GMT +0000). [04:16:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.49, 8.47, 8.40 [04:17:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.84, 8.12, 8.16 [04:19:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.46, 8.43, 8.29 [04:20:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/099f13ff5c17...447205418825 [04:20:28] [02miraheze/ssl] 07MirahezeSSLBot 034472054 - Bot: Update SSL cert for wiki.counterculturelabs.org [04:23:12] PROBLEM - pastport.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pastport.org' expires in 15 day(s) (Thu 31 Mar 2022 03:57:46 GMT +0000). [04:24:53] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.erisly.com' expires in 15 day(s) (Thu 31 Mar 2022 03:59:57 GMT +0000). [04:24:59] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:28:33] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:28:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/447205418825...4dce93e45938 [04:28:51] [02miraheze/ssl] 07MirahezeSSLBot 034dce93e - Bot: Update SSL cert for pastport.org [04:30:14] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Thu 31 Mar 2022 04:02:04 GMT +0000). [04:31:23] PROBLEM - lcn.zfc.id.lv - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lcn.zfc.id.lv' expires in 15 day(s) (Thu 31 Mar 2022 04:07:25 GMT +0000). [04:31:28] RECOVERY - idleon.info - LetsEncrypt on sslhost is OK: OK - Certificate 'idleon.info' will expire on Mon 13 Jun 2022 03:08:16 GMT +0000. [04:34:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4dce93e45938...640c0932b6f1 [04:34:20] [02miraheze/ssl] 07MirahezeSSLBot 03640c093 - Bot: Update SSL cert for wiki.erisly.com [04:35:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/640c0932b6f1...3e33061b79e6 [04:35:03] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Mon 13 Jun 2022 03:12:10 GMT +0000. [04:35:04] [02miraheze/ssl] 07MirahezeSSLBot 033e33061 - Bot: Update SSL cert for podpedia.org [04:35:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3e33061b79e6...c83791dc420e [04:35:46] [02miraheze/ssl] 07MirahezeSSLBot 03c83791d - Bot: Update SSL cert for lcn.zfc.id.lv [04:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:52:48] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Mon 13 Jun 2022 03:02:37 GMT +0000. [04:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:59:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:00:07] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Mon 13 Jun 2022 03:34:58 GMT +0000. [05:00:47] RECOVERY - lcn.zfc.id.lv - LetsEncrypt on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Mon 13 Jun 2022 03:35:39 GMT +0000. [05:09:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:13:40] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Mon 13 Jun 2022 03:20:20 GMT +0000. [05:19:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.55, 1.29 [05:21:12] RECOVERY - pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Mon 13 Jun 2022 03:28:44 GMT +0000. [05:21:57] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.41, 1.27 [05:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:53] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Mon 13 Jun 2022 03:34:11 GMT +0000. [05:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:29:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.20, 1.74, 1.22 [05:31:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.74, 1.70, 1.27 [05:35:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.01, 1.84, 1.43 [05:37:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.86, 1.49 [05:41:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.15, 1.95, 1.60 [05:44:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:27] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.60, 1.29 [05:49:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:51:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.52, 1.30 [05:51:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.65, 2.87 [05:55:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.34, 2.73, 2.71 [05:59:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.64, 1.95, 1.94 [06:05:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.49, 2.04, 1.97 [06:19:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:20:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.52, 7.72, 7.05 [06:22:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.72, 7.58, 7.10 [06:24:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:53:06] [02mediawiki] 07RhinosF1 opened pull request 03#5844: Rel1 38 - 13https://github.com/miraheze/mediawiki/pull/5844 [06:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:01:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:08:26] RhinosF1: good morning 🌞😊 [07:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:19:40] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.79, 1.17 [07:20:34] ugochimobi: it's a better morning [07:21:08] RhinosF1: ah, how's your runny nose? [07:21:19] Calming [07:21:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.45, 1.12 [07:22:24] RhinosF1: Sounds great [07:23:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:35:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:53:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:03:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:23:09] RhinosF1: do well to check my PRs 😊 [08:36:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:37:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.47, 0.93 [08:39:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.47, 1.08, 0.85 [08:49:32] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.38, 1.23 [08:51:30] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.30, 1.22 [08:57:01] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:05] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:27] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:28] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:47] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:08] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:38] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:59:59] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:00:01] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:00:08] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:15] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [09:01:28] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.519 second response time [09:01:34] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:34] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:01:36] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [09:01:37] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [09:01:49] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.221 second response time [09:01:57] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.163 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [09:01:58] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [09:02:11] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.313 second response time [09:03:09] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [09:03:18] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22345 bytes in 1.142 second response time [09:03:18] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.316 second response time [09:03:32] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [09:03:38] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.01817896962 secs [09:11:03] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4296 MB (10% inode=96%); [09:19:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 1.77, 1.37 [09:21:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.56, 1.34 [09:22:32] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [09:42:25] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.36, 0.37, 1.94 [09:46:23] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.30, 0.36, 1.58 [10:07:19] [a58341ae261103c6e69d0044] 2022-03-15 10:06:47: Errore irreversibile di tipo "Exception" i get this error when logging in [10:30:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:59:35] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [10:59:35] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:19:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:20:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 8.17, 7.47 [11:22:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 8.24, 7.58 [11:28:36] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:39:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:46:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:49:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 7.99, 7.47 [11:51:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.90, 7.74, 7.44 [11:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:57:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [11:57:38] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:58:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.16, 2.48 [12:00:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.73, 2.60, 2.35 [12:04:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.99, 8.79, 7.93 [12:06:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.92, 8.38, 7.88 [12:11:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.68, 9.37, 8.04 [12:12:35] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [12:13:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:13:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.09, 8.44, 7.86 [12:14:48] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4294 MB (10% inode=96%); [12:18:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:23:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.82, 1.86, 1.35 [12:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:25:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 8.52, 8.10 [12:26:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.82, 9.23, 8.26 [12:27:32] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:27:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 8.35, 8.04 [12:27:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.39, 1.60, 1.36 [12:28:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 9.35, 8.44 [12:31:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.05, 8.24, 8.20 [12:31:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.27, 8.10, 8.04 [12:31:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.70, 8.36, 8.28 [12:35:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 8.54, 8.40 [12:37:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.97, 8.29, 8.32 [12:40:34] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:49:21] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.65, 1.34 [12:49:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 8.54, 8.16 [12:51:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 8.58, 8.31 [12:51:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 8.02, 7.80 [12:51:19] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.26, 1.47, 1.31 [12:51:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.94, 8.39, 8.17 [12:53:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 9.13, 8.53 [12:53:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 8.02, 7.83 [12:53:59] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.15, 1.86, 1.61 [12:55:59] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.42, 1.74, 1.60 [12:56:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.69, 1.88, 1.21 [12:57:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.90, 9.28, 8.71 [12:57:32] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:57:59] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.52, 1.67, 1.60 [12:58:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.37, 1.32, 1.08 [13:04:59]