[00:00:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.76, 1.48 [00:00:26] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: DISK CRITICAL - free space: / 2304 MB (5% inode=96%); [00:01:01] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.767 second response time [00:02:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.97, 1.60 [00:02:37] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.21, 5.06, 4.73 [00:03:12] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.05, 4.08, 3.50 [00:03:52] RECOVERY - cp21 Disk Space on cp21 is OK: DISK OK - free space: / 9931 MB (25% inode=96%); [00:04:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.69, 1.54 [00:04:26] RECOVERY - cp20 Disk Space on cp20 is OK: DISK OK - free space: / 10248 MB (26% inode=96%); [00:04:37] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.24, 4.66, 4.63 [00:05:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 4.50, 3.75 [00:05:07] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.45, 3.79, 3.46 [00:05:25] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.225 second response time [00:05:36] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.094 second response time [00:08:57] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.21, 3.84, 3.54 [00:10:52] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.39, 3.59, 3.48 [00:12:09] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [00:12:09] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404894590377808 seconds [00:14:42] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.12, 3.32, 3.40 [00:15:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 10.04, 8.71 [00:15:36] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.47, 1.67 [00:17:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.51, 9.57, 8.69 [00:18:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.29, 3.80, 3.57 [00:19:35] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 1.92, 1.81 [00:20:05] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:15] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:34] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.94, 1.83 [00:24:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.83, 1.54 [00:24:42] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.38, 1.64, 1.50 [00:26:12] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.108 second response time [00:27:29] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.320 second response time [00:27:33] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.51, 1.67 [00:27:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.07, 10.49, 9.59 [00:28:03] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:13] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.88, 3.98, 3.81 [00:28:43] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.292 second response time [00:29:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.67, 9.95, 9.51 [00:30:08] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.577 second response time [00:30:17] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.551 second response time [00:30:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.03, 4.13, 3.90 [00:32:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.39, 3.87, 3.83 [00:34:16] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.254 second response time [00:34:17] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [00:36:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.68, 3.93, 3.84 [00:38:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 10.85, 9.99 [00:38:24] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.013 second response time [00:38:28] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.12, 10.33, 9.90 [00:41:17] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [00:42:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.36, 9.71, 9.73 [00:42:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.44, 2.02, 1.75 [00:42:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.91, 3.92, 3.85 [00:42:43] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.474 second response time [00:44:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.82, 1.71 [00:44:40] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.249 second response time [00:46:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.74, 1.45, 1.58 [00:46:56] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.680 second response time [00:48:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.06, 4.16, 3.94 [00:49:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 1.96, 1.72 [00:53:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.92, 1.76 [00:54:21] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:13] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:19] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.168 second response time [00:57:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.93, 1.80 [00:57:37] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.46, 3.92, 3.97 [00:59:42] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.912 second response time [01:00:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.10, 4.07, 4.02 [01:04:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.70, 3.71, 3.89 [01:05:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.94, 1.88 [01:05:24] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.392 second response time [01:06:40] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.209 second response time [01:07:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 9.99, 9.10 [01:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.19, 3.91 [01:09:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.82, 9.76, 9.12 [01:09:26] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:09:28] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.017 second response time [01:10:27] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.443 second response time [01:10:27] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [01:10:27] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407400608062744 seconds [01:11:32] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.273 second response time [01:12:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 3.62, 3.87 [01:15:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.95, 1.89 [01:17:18] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.90, 1.88 [01:18:47] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.246 second response time [01:19:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 2.14, 1.98 [01:20:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.76, 3.88 [01:20:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.01, 4.01, 3.90 [01:20:56] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [01:21:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.03, 11.08, 9.85 [01:21:49] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.329 second response time [01:21:55] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.844 second response time [01:21:59] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.16, 4.16, 3.99 [01:22:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.64, 3.56, 3.76 [01:23:01] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.981 second response time [01:23:01] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.443 second response time [01:23:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 11.00, 9.97 [01:23:59] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.543 second response time [01:24:18] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [01:24:18] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405865669250488 seconds [01:27:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.85, 9.36, 9.59 [01:27:16] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.242 second response time [01:27:21] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:15] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:19] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.091 second response time [01:33:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.90, 1.97 [01:34:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.70, 2.99, 3.34 [01:35:34] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.140 second response time [01:35:38] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.944 second response time [01:38:51] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:04] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.082 second response time [01:39:51] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.258 second response time [01:42:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.40, 3.62, 3.53 [01:44:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.45, 3.97 [01:47:05] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.438 second response time [01:47:08] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.953 second response time [01:47:10] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.81, 2.13, 2.03 [01:48:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.22, 2.86, 3.84 [01:50:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.63, 3.97, 3.68 [01:50:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.32, 3.88 [01:51:06] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:20] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:28] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.33, 3.61 [01:52:16] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:27] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.247 second response time [01:53:53] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [01:54:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.02, 3.46 [01:54:20] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.289 second response time [01:54:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.36, 3.82 [01:56:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.41, 4.24, 3.83 [01:56:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.92, 3.98, 3.80 [01:56:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.53, 4.45, 4.15 [01:57:18] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [02:00:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.16, 3.92, 3.80 [02:01:07] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.63, 1.97 [02:02:26] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.689 second response time [02:02:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.71, 3.81, 3.77 [02:03:26] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 1.057 second response time [02:03:31] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.036 second response time [02:03:44] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.535 second response time [02:06:19] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.254 second response time [02:08:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.52, 4.14, 3.93 [02:09:04] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.35, 1.68 [02:10:39] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.32, 3.89 [02:12:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.46, 3.90, 3.89 [02:12:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.53, 3.26, 3.89 [02:14:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 3.38, 3.82 [02:14:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.31, 4.01, 3.93 [02:15:22] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.23, 2.97, 3.63 [02:19:18] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [02:19:34] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.107 second response time [02:20:16] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.67, 9.33, 7.99 [02:20:57] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.457 second response time [02:21:01] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.75, 1.66 [02:22:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.86, 3.80 [02:22:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.93, 3.93 [02:22:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 4.09, 3.87 [02:22:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.82, 9.40, 8.20 [02:24:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.24, 4.39, 4.10 [02:25:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [02:27:57] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:31] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.606 second response time [02:30:01] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.913 second response time [02:30:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.72, 3.90, 3.97 [02:30:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.52, 1.66, 1.68 [02:32:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.03, 3.90, 3.95 [02:34:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.45, 3.77, 3.90 [02:35:50] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.403 second response time [02:36:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.22, 3.64, 4.00 [02:38:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.87, 3.32, 3.84 [02:39:13] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:16] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.497 second response time [02:42:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.58, 3.87, 3.85 [02:42:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.75, 3.84 [02:43:31] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.255 second response time [02:43:56] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.86, 1.74 [02:44:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 2.60, 3.32 [02:44:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.08, 3.59, 3.75 [02:44:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.40, 3.70 [02:45:29] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.95, 1.78 [02:48:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.15, 2.87, 3.38 [02:49:44] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.19, 3.67, 3.71 [02:52:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.82, 3.49, 3.65 [02:52:46] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.792 second response time [02:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [02:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406226634979248 seconds [02:54:01] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.256 second response time [02:55:41] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.918 second response time [02:55:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [02:55:46] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.880 second response time [02:56:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.82, 3.23, 3.84 [02:56:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.10, 3.89, 3.79 [02:58:23] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:59:56] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.54, 2.72, 1.61 [03:00:31] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:00:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.21, 3.77, 3.78 [03:02:30] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.144 second response time [03:02:34] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.851 second response time [03:03:50] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.90, 1.96 [03:05:50] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.98, 1.99 [03:05:51] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.68, 1.56 [03:07:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.93, 1.97 [03:12:08] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:08] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.54, 2.74, 3.28 [03:13:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.93, 1.95 [03:14:13] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.837 second response time [03:15:47] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.86, 1.92 [03:16:39] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.50, 1.95, 1.70 [03:17:47] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 2.01, 1.97 [03:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.48, 3.43 [03:22:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.80, 1.41, 1.56 [03:22:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 4.02, 3.45 [03:24:38] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.29, 3.16, 3.38 [03:24:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.72, 3.41 [03:26:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.23, 3.27 [03:30:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.50, 2.31, 1.74 [03:30:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.41, 3.07 [03:30:31] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.023 second response time [03:30:36] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:37] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.064 second response time [03:30:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.45, 3.17, 3.29 [03:31:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.81, 2.00 [03:32:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.79, 1.62 [03:32:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 2.92, 2.92 [03:32:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.71, 3.54 [03:34:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 1.90, 1.67 [03:34:31] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.084 second response time [03:34:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.99, 3.21, 3.30 [03:34:41] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.336 second response time [03:34:48] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.166 second response time [03:36:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.92, 1.85, 1.68 [03:36:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 4.31, 3.49 [03:37:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.86, 3.58 [03:38:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.42, 1.54 [03:38:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.56, 3.31 [03:39:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.92, 3.22, 3.38 [03:40:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.23, 3.22 [03:40:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.72, 3.77 [03:41:40] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.41, 1.68 [03:41:42] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [03:43:22] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.66, 3.59, 3.47 [03:45:34] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.46, 3.60 [03:47:12] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.32, 3.85, 3.59 [03:47:28] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.697 second response time [03:48:55] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.236 second response time [03:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.39, 3.20 [03:50:20] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:21] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.51, 3.59 [03:51:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.75, 1.67 [03:51:52] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.454 second response time [03:52:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.30, 3.92, 3.40 [03:52:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.73, 3.65 [03:53:08] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.931 second response time [03:53:36] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.59, 1.62 [03:53:44] Hello [03:54:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.71, 3.38 [03:55:00] I was hoping one may help me in terms of resizing for my icon on the top left of my wiki [03:55:35] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:56:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.83, 3.73 [03:56:38] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.922 second response time [03:56:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 4.09, 3.53 [03:56:48] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.80, 3.88, 3.82 [03:57:26] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:00:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.37, 3.33 [04:00:39] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.09, 3.92, 3.84 [04:00:43] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.013 second response time [04:00:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.84, 3.56 [04:01:29] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.221 second response time [04:01:41] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.756 second response time [04:02:34] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.478 second response time [04:02:45] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.643 second response time [04:04:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 2.89, 3.35 [04:04:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.09, 2.92, 3.25 [04:07:57] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:09:55] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 1.019 second response time [04:10:39] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.84, 3.91, 3.88 [04:12:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.21, 4.00, 3.92 [04:16:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.61, 1.31 [04:16:12] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.321 second response time [04:17:30] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.016 second response time [04:18:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.34, 1.24 [04:18:21] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.274 second response time [04:18:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.32, 3.95, 3.98 [04:19:28] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 1.85, 1.61 [04:20:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.74, 4.13, 4.03 [04:21:37] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.622 second response time [04:22:30] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.062 second response time [04:22:33] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.468 second response time [04:23:27] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.92, 1.72 [04:23:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [04:24:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.80, 3.71, 3.91 [04:25:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.99, 1.77 [04:25:51] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.018 second response time [04:27:26] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.98, 1.80 [04:28:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.46, 3.31 [04:28:58] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:18] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:54] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.360 second response time [04:30:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.46, 3.31 [04:30:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.14, 3.88, 3.91 [04:31:19] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.013 second response time [04:31:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 2.00, 1.84 [04:33:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.87, 1.81 [04:34:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.49, 3.36 [04:34:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.57, 3.92, 3.93 [04:36:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.72, 4.07, 3.57 [04:37:17] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.549 second response time [04:37:22] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.867 second response time [04:37:32] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.402 second response time [04:38:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.41, 3.39 [04:39:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.81, 1.78 [04:41:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.88, 1.81 [04:43:21] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 2.02, 1.87 [04:45:21] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.96, 1.87 [04:46:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.81, 3.58 [04:47:20] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 2.05, 1.91 [04:48:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.21, 3.47, 3.51 [04:50:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.84, 4.00, 3.78 [04:51:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 2.48, 1.38 [04:52:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.35, 3.18, 3.37 [04:52:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.89, 3.77 [04:52:55] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.020 second response time [04:53:01] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [04:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40694785118103 seconds [04:53:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.56, 1.79, 1.26 [04:54:13] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:49] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 0.458 second response time [04:55:07] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.242 second response time [04:55:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.40, 1.68, 1.28 [04:57:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.31, 3.55 [04:57:23] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:58:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.52, 4.62, 3.52 [04:58:26] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.660 second response time [04:58:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 4.57, 3.95 [05:01:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.66, 3.46 [05:02:36] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:02:36] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:39] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.577 second response time [05:05:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.84, 1.96 [05:06:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 4.03, 3.66 [05:07:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.92, 1.97 [05:08:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.57, 3.54 [05:10:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 3.93, 3.66 [05:10:56] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.537 second response time [05:11:01] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.581 second response time [05:11:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.86, 1.95 [05:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.68, 3.76 [05:12:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.85, 3.67 [05:14:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.08, 3.73, 3.58 [05:16:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.91, 3.74, 3.60 [05:18:14] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [05:18:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.32, 4.00, 3.72 [05:18:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.54, 4.27, 3.82 [05:18:45] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.73, 2.01, 1.92 [05:20:03] !log reception@mwtask111:~/criticalrole$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=criticalrolewiki --search-recursively /home/reception/criticalrole --user=Miraheze_maintenance_script --comment="Imported from criticalrole.fandom.com" [05:20:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 4.08, 3.82 [05:20:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:20:47] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.096 second response time [05:23:11] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.242 second response time [05:23:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.26, 1.76, 1.86 [05:23:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [05:23:54] !log RENAMED degrowthgreensaotearoawiki to anzdegrowthtransitionwiki. [05:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:11] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:34] !log RENAMED kujouwiki to kawasakiwiki [05:25:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:27] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.022 second response time [05:28:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.27, 3.86, 3.87 [05:29:10] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.45, 1.68 [05:29:23] !log DELETED and DRIOPPED ahmogleewiki (reset, T9242) [05:29:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:32] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [05:32:36] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.015 second response time [05:32:39] Reception123: heh can always tell when you're back from a wikibreak due to the number of scripts that end up being run...anyway, thanks for remembering to set the username for the `ImportImages.php` mwscript to "User:Miraheze maintenance script" :) [05:34:23] dmehus: heh yes, it seems like I'm the only one who does them [05:34:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.28, 4.06, 3.93 [05:35:29] Reception123: heh yeah though in fairness, you're the only MWE team member capable of doing the database renames [05:36:39] yeah, those definitely [05:36:56] !log DELETED and DROPPED pokemoncometwiki (reset, T9234) [05:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:37:50] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.635 second response time [05:38:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.70, 3.99, 3.95 [05:39:15] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 0.685 second response time [05:39:32] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.769 second response time [05:39:41] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.584 second response time [05:41:50] !log RENAMED cnocbridewiki to smithwickswiki [05:41:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.15, 3.31, 3.93 [05:46:10] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=sotfwiki --search-recursively /home/reception/sotfwiki --user=Miraheze_maintenance_script --comment="Imported from sotf.fandom.com" [05:46:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:48:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.31, 3.79, 3.89 [05:48:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.53, 3.88, 3.90 [05:49:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 1.83, 1.59 [05:51:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.84, 1.63 [05:51:45] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:26] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:27] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.021 second response time [05:54:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.67, 3.85 [05:55:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.58, 1.69, 1.62 [05:57:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.80, 1.35, 1.00 [05:59:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.92, 1.15, 0.97 [06:00:47] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.344 second response time [06:01:58] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.493 second response time [06:04:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.58, 3.80, 3.91 [06:04:42] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.015 second response time [06:04:55] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:41] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [06:06:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.36, 4.02, 3.97 [06:06:45] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.624 second response time [06:06:54] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.887 second response time [06:07:00] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.292 second response time [06:08:36] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.16, 2.32, 1.87 [06:10:40] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.398 second response time [06:10:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.43, 3.43, 3.93 [06:12:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.91, 2.78, 3.34 [06:12:15] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:12:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.39, 3.83, 3.93 [06:16:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 2.61, 3.38 [06:18:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.81, 3.88, 3.88 [06:20:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.93, 3.84, 3.86 [06:22:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.09, 3.94, 3.89 [06:22:58] We do apologise for any issues [06:23:12] The job queue is broken and Matomo very upset about it's data [06:24:23] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:17] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.251 second response time [06:25:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.83, 1.32, 1.10 [06:26:24] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type=htmlCacheUpdate (START) [06:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:39] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.018 second response time [06:27:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.03, 1.21, 1.08 [06:28:29] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.819 second response time [06:28:33] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.419 second response time [06:28:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.72, 3.43 [06:28:44] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.357 second response time [06:28:44] [02puppet] 07Universal-Omega commented on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1135462870 [06:28:45] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [06:30:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.98, 3.47, 2.82 [06:30:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.29, 3.87, 3.92 [06:30:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.29, 3.40, 3.34 [06:31:44] [02puppet] 07RhinosF1 commented on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1135464943 [06:31:44] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [06:32:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.28, 3.78, 3.01 [06:33:02] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.20, 4.18, 3.60 [06:36:31] PROBLEM - gluster101 Disk Space on gluster101 is WARNING: DISK WARNING - free space: / 87988 MB (10% inode=80%); [06:36:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 4.00, 3.58 [06:38:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.80, 3.62 [06:38:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.50, 3.44 [06:40:18] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.019 second response time [06:42:20] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.391 second response time [06:43:45] !log killed an import [06:43:48] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type=htmlCacheUpdate (END - exit=2) [06:43:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:55] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (START) [06:44:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:44:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.07, 3.00, 3.28 [06:46:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.71, 3.97, 3.72 [06:46:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.75, 4.07, 3.71 [06:46:28] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:46] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:48:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.95, 3.90, 3.59 [06:49:35] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.920 second response time [06:49:41] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.543 second response time [06:50:08] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.15, 3.95, 3.76 [06:50:21] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.845 second response time [06:50:24] mwtask load is expected [06:50:41] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 0.939 second response time [06:52:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.89, 3.80 [06:52:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.38, 3.85, 3.71 [06:52:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.76, 3.61 [06:53:15] RhinosF1: high jobqueue on the wikis in import always happens during an import it catches up after awhile, but I don't think that does happen if --no-updates is used, which should always be done for imports. Anyways, I'm not going to be around for awhile and have already told Reception123 the reason, so please leave me out of anything further for now. [06:54:18] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:54:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.73, 3.68 [06:56:08] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.49, 3.91, 3.84 [06:56:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.22, 3.11, 3.39 [06:58:00] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 2.78, 3.31 [06:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 2.90, 3.81 [06:58:44] CosmicAlpha: ok [07:00:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.37, 3.87 [07:02:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.05, 3.68 [07:04:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.38, 2.87, 3.35 [07:04:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.15, 3.61, 3.61 [07:06:20] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:06:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.01, 2.43, 3.29 [07:10:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.61, 3.76, 3.69 [07:11:21] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:08] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.179 second response time [07:14:22] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:51] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.350 second response time [07:15:31] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.199 second response time [07:17:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.64, 2.46, 1.81 [07:18:28] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:18:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.90, 4.30, 3.91 [07:19:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.67, 1.86, 1.67 [07:19:32] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.018 second response time [07:21:37] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.676 second response time [07:22:44] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.028 second response time [07:22:44] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.148 second response time [07:23:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.24, 1.56, 1.59 [07:26:59] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:06] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:11] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.292 second response time [07:32:51] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.061 second response time [07:38:12] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.092 second response time [07:39:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.43, 3.03 [07:40:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.34, 1.78, 1.31 [07:41:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 3.53, 3.09 [07:42:11] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.91, 1.74, 1.34 [07:43:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.56, 3.15 [07:43:27] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:06] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.12, 1.50, 1.30 [07:44:17] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.219 second response time [07:45:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.60, 3.20 [07:47:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.49, 3.20 [07:47:22] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.142 second response time [07:47:41] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.177 second response time [07:48:55] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.050 second response time [07:48:58] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:50] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:18] [02puppet] 07Reception123 commented on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1135532907 [07:53:18] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [07:53:57] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.357 second response time [07:54:00] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:56:07] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.982 second response time [07:56:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.06, 3.16 [08:01:32] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.310 second response time [08:06:27] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:33] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.349 second response time [08:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.30, 3.10 [08:11:15] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.586 second response time [08:12:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 2.94, 3.00 [08:16:33] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.25, 3.60, 3.97 [08:20:31] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.66, 1.52 [08:20:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.77, 3.96, 4.05 [08:21:04] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.248 second response time [08:21:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.51, 3.02 [08:21:24] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.016 second response time [08:22:28] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.66, 1.64, 1.53 [08:22:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.90, 1.88, 1.57 [08:22:43] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.339 second response time [08:23:21] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [08:23:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406412839889526 seconds [08:24:15] !log restart criticalrolewiki import with --sleep enabled [08:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:24] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.65, 1.52 [08:24:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.67, 2.81, 2.86 [08:25:11] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 1.213 second response time [08:25:22] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.340 second response time [08:25:26] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.072 second response time [08:26:20] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.96, 2.23, 1.75 [08:28:17] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.90, 1.68 [08:28:34] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:16] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.494 second response time [08:29:27] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:37] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.035 second response time [08:30:15] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.38, 1.96, 1.72 [08:30:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.70, 4.34, 3.62 [08:30:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.89, 3.31 [08:31:40] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.783 second response time [08:32:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.26, 1.70, 1.66 [08:32:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.86, 3.55 [08:32:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.68, 3.32 [08:34:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.89, 3.96, 4.00 [08:34:43] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.611 second response time [08:36:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.95, 4.12, 4.04 [08:36:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.17, 3.19 [08:38:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.01, 3.30 [08:38:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.38, 3.77, 3.92 [08:40:39] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.81, 2.90, 2.03 [08:42:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 3.92, 3.60 [08:42:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.15, 3.97, 3.96 [08:42:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 3.76, 3.39 [08:43:34] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.84, 3.47 [08:45:31] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.58, 1.89, 1.83 [08:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.99, 3.75 [08:46:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.39, 3.75, 3.88 [08:46:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.90, 3.21, 3.28 [08:47:34] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.494 second response time [08:50:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.28, 1.49, 1.67 [08:50:24] [02mw-config] 07ugochimobi opened pull request 03#4710: add Kartographer config to ManageWiki - 13https://github.com/miraheze/mw-config/pull/4710 [08:50:24] [url] Page not found · GitHub · GitHub | github.com [08:50:38] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.40, 3.89, 3.88 [08:51:32] miraheze/mw-config - ugochimobi the build passed. [08:52:19] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.18, 2.86, 3.32 [08:53:29] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.231 second response time [08:57:33] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.236 second response time [08:57:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.74, 1.69 [08:58:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.43, 1.96 [08:58:25] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.881 second response time [08:59:37] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.899 second response time [08:59:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.47, 1.68, 1.67 [09:04:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.12, 1.22, 1.69 [09:05:58] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.007 second response time [09:06:04] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.463 second response time [09:06:18] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.020 second response time [09:10:17] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.271 second response time [09:10:41] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [09:11:26] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:12:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.69, 1.52 [09:14:27] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:26] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.58, 1.54 [09:16:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.60, 1.59 [09:18:18] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.075 second response time [09:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 1.86, 1.69 [09:18:29] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.105 second response time [09:18:39] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.233 second response time [09:21:26] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.967 second response time [09:22:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.93, 1.76 [09:22:44] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.96, 1.70, 1.49 [09:22:50] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.017 second response time [09:22:51] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:39] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.22, 2.32, 1.74 [09:26:57] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.307 second response time [09:27:03] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.743 second response time [09:27:42] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.85, 2.25, 1.77 [09:28:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.61, 1.67 [09:31:57] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.509 second response time [09:33:18] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.253 second response time [09:34:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.09, 1.75, 1.82 [09:35:23] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:06] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.09, 1.46, 1.70 [09:36:50] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880287046 [09:36:51] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [09:37:27] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.070 second response time [09:37:33] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.614 second response time [09:38:12] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.394 second response time [09:40:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.86, 1.74 [09:40:59] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.55, 1.96, 1.98 [09:42:22] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.852 second response time [09:42:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.93, 1.78 [09:44:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 2.13, 1.87 [09:45:03] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.876 second response time [09:45:52] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880296087 [09:45:53] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [09:46:14] [02puppet] 07RhinosF1 commented on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1135673437 [09:46:14] ... [09:46:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.98, 1.85 [09:47:22] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:20] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880296087 [09:48:20] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [09:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 2.03, 1.88 [09:49:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.26, 1.86, 1.87 [09:51:23] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.347 second response time [09:51:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.36, 1.66, 1.80 [09:52:51] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.035 second response time [09:53:01] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.254 second response time [09:53:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.20, 2.01, 1.69 [09:54:55] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.323 second response time [09:55:08] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [09:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:58:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.98, 1.97 [09:59:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.31, 1.54, 1.70 [10:03:25] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.43, 3.03 [10:06:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.04, 3.59, 3.94 [10:06:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.58, 2.70, 2.80 [10:08:15] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.010 second response time [10:08:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 2.05, 1.98 [10:09:26] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.234 second response time [10:09:38] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 2.389 second response time [10:10:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.09, 3.74, 3.91 [10:11:10] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:39] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.55, 3.77, 3.89 [10:14:43] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880324337 [10:14:45] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [10:15:42] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.867 second response time [10:15:42] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [10:15:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.05, 1.34, 1.91 [10:18:39] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.11, 3.73, 3.83 [10:19:49] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.97, 3.83, 3.85 [10:20:51] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.70, 1.05, 1.61 [10:23:45] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.037 second response time [10:26:05] Reception123: you might want to runJobs again [10:26:09] It's still a tad high [10:26:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.80, 1.96 [10:26:42] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.537 second response time [10:26:47] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.584 second response time [10:27:27] Reception123: someone is triggering lots of echo notifications [10:27:33] Which is holding the queue up [10:27:46] hmm, I've used sleep this time.. [10:27:51] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.132 second response time [10:27:51] Either that or it's still too busy [10:27:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.76, 1.72 [10:28:00] Reception123: check how many jobs are on that wiki [10:28:03] Try clear them [10:28:08] Then runJobs across the farm [10:28:20] probably running the html ones is the best [10:28:37] Reception123: half the jobs are from echo [10:28:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.57, 4.45, 4.10 [10:28:42] I think it's right on the line [10:28:58] hmm, I guess all of them then [10:29:32] Please [10:29:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.08, 1.47, 1.62 [10:30:05] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.780 second response time [10:30:33] RhinosF1: how does foreachwikiindblist work with mwscript? [10:32:37] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php [10:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:34:23] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.762 second response time [10:34:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.04, 1.34, 1.68 [10:36:08] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:43] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.84, 1.61 [10:38:23] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.81, 1.97, 1.73 [10:42:38] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.75, 3.64, 3.93 [10:43:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.04, 1.94, 1.75 [10:44:37] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.47, 4.34, 4.14 [10:45:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.74, 1.83, 1.73 [10:47:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.47, 2.89 [10:47:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.29, 1.57, 1.64 [10:48:22] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.807 second response time [10:48:30] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.512 second response time [10:49:57] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.276 second response time [10:51:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.56, 3.09 [10:51:14] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:03] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.671 second response time [10:53:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.01, 3.27, 3.03 [10:53:16] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15616 MB (11% inode=99%); [10:55:42] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [10:55:54] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:29] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.76, 1.69 [10:57:14] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.41, 1.68, 1.47 [10:57:24] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.358 second response time [10:58:26] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.20, 1.58, 1.64 [10:59:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.63, 1.58, 1.46 [11:00:35] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 3.58, 3.15 [11:01:49] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:31] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:48] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:03:49] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 3.659 second response time [11:04:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.31, 1.79, 1.73 [11:06:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.16, 1.58, 1.66 [11:06:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.35, 3.23 [11:06:39] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [11:06:41] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.985 second response time [11:06:41] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 5.041 second response time [11:06:48] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.252 second response time [11:08:38] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.124 second response time [11:09:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.59, 2.11, 1.72 [11:10:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.98, 1.72, 1.69 [11:12:02] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.09, 1.85, 1.74 [11:13:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.64, 1.89, 1.73 [11:13:56] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:59] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.83, 1.74 [11:14:00] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.044 second response time [11:14:24] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.04, 1.65, 1.66 [11:15:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.25, 2.03, 1.82 [11:16:29] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 8.351 second response time [11:16:42] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.015 second response time [11:17:53] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.97, 1.83 [11:18:09] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 7.441 second response time [11:19:51] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.18, 1.96, 1.84 [11:20:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.71, 2.05, 1.74 [11:20:34] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.142 second response time [11:21:48] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.97, 1.84, 1.80 [11:22:35] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:20] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.13, 2.62, 2.08 [11:25:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.71, 1.64 [11:26:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.45, 3.20 [11:26:24] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.327 second response time [11:26:30] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 9.281 second response time [11:26:46] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 6.738 second response time [11:27:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.97, 2.89, 3.03 [11:29:25] Reception123: mwscript script.php all [11:30:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 2.00, 1.89 [11:30:40] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.957 second response time [11:30:41] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 302 Found - 354 bytes in 4.909 second response time [11:31:45] Reception123: looks far better now [11:31:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.63, 1.65 [11:32:15] It looks ok since :49 [11:32:27] ah ok, thanks [11:32:28] and yeah [11:33:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.34, 1.88, 1.96 [11:37:09] !log matomo101: ./console core:update [11:37:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.97, 1.91 [11:41:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/75583be9c78f...c3ec35362c3c [11:41:06] [url] Comparing 75583be9c78f...c3ec35362c3c · miraheze/puppet · GitHub | github.com [11:41:07] [02miraheze/puppet] 07paladox 03c3ec353 - matomo101: increase php::fpm::fpm_workers_multiplier to 5.0 [11:42:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.93, 1.91 [11:42:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.17, 3.25, 3.91 [11:43:12] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [11:43:12] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:47:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.41, 1.43, 1.69 [11:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 2.12, 1.99 [11:50:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.38, 2.64, 3.36 [11:51:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.64, 1.92, 1.82 [11:53:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.29, 1.72, 1.77 [11:57:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.17, 1.99, 1.87 [11:59:09] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.98, 1.90, 1.63 [11:59:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 3.92, 3.20 [11:59:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.34, 4.39, 3.19 [11:59:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.60, 1.86, 1.84 [12:00:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.30, 3.57, 2.72 [12:01:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.63, 3.19 [12:01:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.77, 3.11 [12:02:58] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.11, 1.50, 1.52 [12:03:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 4.00, 3.27 [12:03:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.52, 1.70 [12:04:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.61, 2.92 [12:04:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.92, 1.98 [12:05:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.95, 3.41 [12:05:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.87, 3.31 [12:06:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.70, 3.03 [12:06:40] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [12:06:40] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4079813957214355 seconds [12:07:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.85, 3.45 [12:07:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.57, 1.86, 1.83 [12:08:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 3.08, 2.88 [12:09:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 3.26, 3.20 [12:09:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.77, 1.44, 1.67 [12:11:46] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:14:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 4.08, 3.62 [12:15:22] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.51, 1.75, 1.63 [12:17:17] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 1.73, 1.63 [12:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.83, 1.85 [12:18:31] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.91, 1.77 [12:18:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.74, 3.58 [12:19:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.67, 1.62 [12:20:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.84, 3.63 [12:21:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.30, 3.04 [12:22:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.40, 3.50 [12:22:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.36, 3.18, 3.02 [12:23:01] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.33, 1.76, 1.69 [12:23:33] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [12:24:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.94, 1.92 [12:24:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 1.86, 1.73 [12:28:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.18, 1.86, 1.89 [12:28:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.10, 3.35 [12:30:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.42, 1.48, 1.70 [12:30:39] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.93, 1.64, 1.71 [12:34:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.62, 2.79, 1.54 [12:34:28] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.16, 1.96, 1.81 [12:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.17, 10.57, 9.00 [12:36:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.86, 1.98, 1.39 [12:36:39] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [12:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.26, 10.73, 9.27 [12:38:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.85, 2.24, 2.00 [12:38:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.54, 1.46, 1.27 [12:38:17] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.62, 1.97, 1.86 [12:40:02] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.87, 1.89 [12:40:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [12:40:20] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:44:01] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.47, 1.54, 1.69 [12:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 4.19, 3.53 [12:44:37] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [12:45:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.02, 9.86, 9.56 [12:47:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 1.25, 1.61 [12:50:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.92, 1.66 [12:50:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.52, 3.14 [12:50:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.88, 3.25 [12:51:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.56, 1.92, 1.77 [12:52:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 3.27, 3.10 [12:52:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.49, 3.17 [12:54:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.86, 1.71 [12:54:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.83, 3.83 [12:54:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.01, 2.06, 1.83 [12:54:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 2.99, 3.02 [12:58:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.62, 1.65 [12:59:12] RhinosF1: images aren't rendering correctly in gpcommonswiki [12:59:29] I thought it was my network [12:59:35] but it clearly isn't [13:01:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.42, 1.85, 1.89 [13:06:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.44, 3.53 [13:07:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.73, 1.67 [13:08:54] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:09:21] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.88, 1.73 [13:10:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.89, 1.99 [13:10:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.34, 3.47 [13:10:37] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:12:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.53, 3.51 [13:14:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.09, 3.17, 3.37 [13:15:20] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.99, 1.83 [13:17:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.35, 2.06, 1.87 [13:17:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.10, 1.45, 1.64 [13:18:07] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.12, 2.27, 2.08 [13:22:02] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.94, 1.97 [13:23:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [13:23:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407023191452026 seconds [13:23:59] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.46, 2.22, 2.08 [13:25:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.11, 2.22, 1.87 [13:29:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.69, 4.52, 3.47 [13:30:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.52, 5.04, 3.98 [13:30:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 4.22, 3.28 [13:31:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.91, 1.96 [13:33:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.18, 1.72, 1.88 [13:36:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.84, 3.49 [13:36:40] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [13:36:40] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403776407241821 seconds [13:37:40] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.64, 1.81, 2.00 [13:39:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.35, 2.05, 2.00 [13:39:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.83, 3.01, 2.41 [13:40:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.54, 3.71, 3.77 [13:41:13] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.97, 1.98 [13:42:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 3.29, 3.39 [13:43:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.96, 1.33, 1.62 [13:45:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.97, 1.98 [13:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.44, 3.89 [13:46:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 2.84, 3.36 [13:51:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.66, 9.05, 8.38 [13:53:38] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.96, 9.13, 8.48 [13:54:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.33, 3.59 [13:57:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.07, 1.87, 1.67 [14:01:07] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.77, 1.96 [14:07:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.41, 3.33 [14:07:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.48, 2.00, 1.95 [14:08:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.89, 3.94 [14:09:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 2.03, 1.96 [14:10:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 4.11, 4.00 [14:11:05] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.42, 1.67 [14:11:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.23, 3.26 [14:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.60, 3.83 [14:13:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.48, 1.61, 1.94 [14:14:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.99, 3.96 [14:16:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.85, 3.93 [14:19:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.68, 1.64 [14:19:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.10, 1.66, 1.82 [14:20:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.83, 3.88 [14:21:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.18, 1.74, 1.93 [14:23:01] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.75, 1.68 [14:23:32] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [14:25:01] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.65, 1.64, 1.64 [14:25:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.12, 1.90, 1.94 [14:26:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.41, 3.77 [14:28:59] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.80, 1.70 [14:30:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.71, 1.68 [14:31:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.05, 1.92, 1.99 [14:32:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.53, 1.59, 1.63 [14:33:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.08, 2.68, 2.26 [14:34:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.96, 3.79 [14:36:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.27, 2.97 [14:38:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 2.91, 2.87 [14:38:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.98, 3.87 [14:40:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.65, 5.55, 4.47 [14:42:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.64, 3.20 [14:42:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.76, 3.43 [14:43:16] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14590 MB (10% inode=99%); [14:44:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.21, 3.09 [14:44:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 3.17, 3.25 [14:45:16] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15604 MB (11% inode=99%); [14:48:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.49, 3.23 [14:48:53] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.94, 2.07, 1.76 [14:50:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.20, 3.16 [14:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [14:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.410457611083984 seconds [15:00:51] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.70, 1.98, 1.96 [15:02:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.54, 3.94 [15:03:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.93, 1.82, 1.99 [15:06:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 4.22, 4.12 [15:09:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.51, 1.74, 2.00 [15:09:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.01, 1.30, 1.69 [15:12:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.64, 3.91 [15:14:47] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.93, 1.91 [15:16:46] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.91, 1.91 [15:17:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.29, 1.35, 1.70 [15:17:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.78, 1.77 [15:18:46] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 2.14, 2.00 [15:21:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.27, 1.57, 1.68 [15:24:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.37, 3.53 [15:25:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.05, 1.92, 1.75 [15:26:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.12, 3.42 [15:27:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.72, 1.70 [15:27:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.23, 1.95, 1.78 [15:28:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.13, 3.39 [15:29:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.37, 1.77, 1.74 [15:31:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.31, 1.60, 1.67 [15:31:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.12, 1.55, 1.66 [15:34:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.81, 3.54 [15:35:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.31, 1.96, 1.81 [15:37:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.38, 1.72, 1.74 [15:38:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.33, 3.43 [15:39:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.21, 1.89, 1.74 [15:39:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.06, 1.50, 1.65 [15:40:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.44, 3.43 [15:40:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.78, 1.94 [15:41:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.88, 1.82, 1.73 [15:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.57, 3.49 [15:43:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.54, 1.61, 1.66 [15:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.97, 3.64 [15:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.56, 3.53 [15:48:40] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.81, 1.85 [15:50:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.23, 3.40 [15:51:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.21, 1.80, 1.71 [15:52:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.36, 1.79, 1.85 [15:53:32] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [15:55:25] [02puppet] 07Universal-Omega synchronize pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [15:55:26] [url] Page not found · GitHub · GitHub | github.com [15:55:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.85, 1.76 [15:55:47] [02puppet] 07Universal-Omega synchronize pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [15:55:47] [url] Page not found · GitHub · GitHub | github.com [15:58:11] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Thu 09 Jun 2022 15:37:28 GMT +0000). [15:58:18] PROBLEM - datacrondatabase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datacrondatabase.com' expires in 15 day(s) (Thu 09 Jun 2022 15:33:39 GMT +0000). [15:58:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.58, 3.43 [15:58:37] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.06, 1.38, 1.66 [15:59:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.35, 1.65, 1.70 [15:59:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.91, 2.19, 1.91 [16:00:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.46, 3.40 [16:01:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.36, 1.87, 1.82 [16:02:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.49, 3.40 [16:03:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ea68a7dd0f2a...73cf1d73bd3c [16:03:10] [url] Comparing ea68a7dd0f2a...73cf1d73bd3c · miraheze/ssl · GitHub | github.com [16:03:11] [02miraheze/ssl] 07MirahezeSSLBot 0373cf1d7 - Bot: Update SSL cert for mwcosmos.com [16:03:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 1.99, 1.87 [16:04:02] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Thu 09 Jun 2022 15:37:28 GMT +0000). [16:04:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 3.28, 3.33 [16:05:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.88, 1.84 [16:07:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.85, 1.77 [16:09:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.32, 1.62, 1.69 [16:10:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/73cf1d73bd3c...76347ce3cba3 [16:10:37] [url] Comparing 73cf1d73bd3c...76347ce3cba3 · miraheze/ssl · GitHub | github.com [16:10:37] [02miraheze/ssl] 07MirahezeSSLBot 0376347ce - Bot: Update SSL cert for datacrondatabase.com [16:12:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.97, 9.68, 7.53 [16:13:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.89, 1.82 [16:14:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.71, 10.59, 8.15 [16:16:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.40, 4.22, 3.70 [16:16:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.89, 9.40, 8.00 [16:16:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.94, 3.44, 3.07 [16:17:23] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bluearchive.wiki' expires in 15 day(s) (Thu 09 Jun 2022 15:45:47 GMT +0000). [16:17:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.32, 2.17, 1.95 [16:17:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.22, 1.48, 1.67 [16:18:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.94, 2.89, 2.91 [16:19:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.17, 1.82, 1.85 [16:20:30] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Thu 09 Jun 2022 16:18:00 GMT +0000). [16:20:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 2.17, 1.80 [16:21:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.44, 2.46, 2.07 [16:21:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.10, 2.24, 1.92 [16:22:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.14, 4.07, 3.35 [16:22:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/76347ce3cba3...24b1a080de61 [16:22:53] [url] Comparing 76347ce3cba3...24b1a080de61 · miraheze/ssl · GitHub | github.com [16:22:54] [02miraheze/ssl] 07MirahezeSSLBot 0324b1a08 - Bot: Update SSL cert for bluearchive.wiki [16:23:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [16:23:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404934883117676 seconds [16:24:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.85, 3.36 [16:24:30] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.91, 1.79 [16:24:37] PROBLEM - factorio-mods.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'factorio-mods.tk' expires in 15 day(s) (Thu 09 Jun 2022 16:08:55 GMT +0000). [16:27:23] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [16:27:36] RECOVERY - datacrondatabase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'datacrondatabase.com' will expire on Mon 22 Aug 2022 15:10:26 GMT +0000. [16:28:29] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.48, 1.60, 1.69 [16:29:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.41, 3.21 [16:31:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.67, 3.85 [16:31:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.23, 3.13, 3.11 [16:32:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.65, 3.12, 3.22 [16:32:27] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 1.77, 1.74 [16:32:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/24b1a080de61...95ba9d48f861 [16:32:34] [url] Comparing 24b1a080de61...95ba9d48f861 · miraheze/ssl · GitHub | github.com [16:32:35] [02miraheze/ssl] 07MirahezeSSLBot 0395ba9d4 - Bot: Update SSL cert for factorio-mods.tk [16:33:34] RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [16:35:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.96, 3.93 [16:36:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.96, 1.81 [16:36:39] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [16:39:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.82, 3.90 [16:40:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [16:41:52] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fortis.wiki' expires in 15 day(s) (Thu 09 Jun 2022 16:18:00 GMT +0000). [16:42:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/95ba9d48f861...a09e72cb77d2 [16:42:22] [url] Comparing 95ba9d48f861...a09e72cb77d2 · miraheze/ssl · GitHub | github.com [16:42:23] [02miraheze/ssl] 07MirahezeSSLBot 03a09e72c - Bot: Update SSL cert for fortis.wiki [16:43:13] [02puppet] 07Universal-Omega synchronize pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [16:43:13] [url] Page not found · GitHub · GitHub | github.com [16:43:52] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4272 MB (10% inode=96%); [16:44:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.46, 4.91, 4.22 [16:45:52] [02puppet] 07Universal-Omega synchronize pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [16:45:52] [url] Page not found · GitHub · GitHub | github.com [16:46:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.89, 1.89 [16:46:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.55, 3.27 [16:46:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.84, 3.40 [16:47:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.35, 1.71, 2.00 [16:48:10] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.87, 7.24, 5.95 [16:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.64, 2.20, 2.01 [16:48:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.43, 3.31 [16:50:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.54, 6.54, 5.84 [16:50:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.11, 3.19 [16:51:54] [02puppet] 07Universal-Omega reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880746076 [16:51:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.82, 1.98 [16:51:55] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [16:52:05] [02puppet] 07Universal-Omega reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880746242 [16:52:05] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [16:52:17] [02puppet] 07Universal-Omega reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880746450 [16:52:18] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [16:52:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 3.15, 3.22 [16:56:42] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880750148 [16:56:43] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [16:56:43] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880748223 [16:56:44] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [16:56:45] [02puppet] 07RhinosF1 reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880748680 [16:56:45] ... [16:59:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.88, 3.49 [17:01:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.71, 3.47 [17:02:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.84, 1.96 [17:02:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.67, 10.33, 9.57 [17:03:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.76, 3.51 [17:04:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.14, 10.04, 9.55 [17:05:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.60, 3.35 [17:05:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.64, 3.50 [17:06:40] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [17:06:40] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40669584274292 seconds [17:06:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 3.31, 3.29 [17:07:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.27, 3.39 [17:08:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.52, 9.85 [17:09:02] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:11:14] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [17:11:52] [02puppet] 07Universal-Omega synchronize pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [17:11:53] [url] Page not found · GitHub · GitHub | github.com [17:12:02] [02puppet] 07Universal-Omega reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880764028 [17:12:02] ... [17:12:21] [02puppet] 07Universal-Omega reviewed pull request 03#2587 commit - 13https://github.com/miraheze/puppet/pull/2587#discussion_r880764296 [17:12:22] ... [17:14:19] [02puppet] 07Universal-Omega synchronize pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587 [17:14:19] [url] Page not found · GitHub · GitHub | github.com [17:14:26] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4267 MB (10% inode=96%); [17:14:55] [02puppet] 07RhinosF1 commented on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1136213661 [17:14:56] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [17:15:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.47, 4.38, 3.75 [17:15:23] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [17:15:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.60, 10.11, 10.01 [17:15:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.67, 1.82, 2.00 [17:16:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 4.11, 3.68 [17:16:47] [02puppet] 07RhinosF1 edited a comment on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1136213661 [17:16:49] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [17:18:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.21, 3.45, 3.49 [17:18:31] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [17:18:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.57, 3.56 [17:20:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.48, 3.48 [17:20:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 2.01, 1.91 [17:22:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.18, 3.37 [17:22:22] [02puppet] 07Universal-Omega commented on pull request 03#2587: deploy-mediawiki: add arguments for branch and rebuildLC language - 13https://github.com/miraheze/puppet/pull/2587#issuecomment-1136228461 [17:22:23] [url] deploy-mediawiki: add arguments for branch and rebuildLC language by Universal-Omega · Pull Request #2587 · miraheze/puppet · GitHub | github.com [17:22:37] RECOVERY - factorio-mods.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'factorio-mods.tk' will expire on Mon 22 Aug 2022 15:32:28 GMT +0000. [17:23:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.42, 10.69, 10.24 [17:24:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.84, 1.88 [17:24:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.10, 2.95, 3.34 [17:27:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.17, 1.79, 1.86 [17:28:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.26, 9.69, 9.20 [17:29:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.16, 1.56, 1.77 [17:30:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.46, 9.79, 9.31 [17:30:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.40, 3.47 [17:31:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 10.09, 10.17 [17:31:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.95, 1.32, 1.65 [17:32:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.09, 3.34 [17:34:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.13, 3.90 [17:36:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.36, 4.28, 4.20 [17:38:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.54, 1.70 [17:40:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.51, 3.89 [17:42:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.92, 4.00 [17:50:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 1.91, 1.74 [17:54:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.84, 4.66, 3.68 [17:54:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.93, 1.81 [17:55:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.36, 4.18, 3.69 [17:56:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.80, 3.48 [17:56:33] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.12, 1.91, 1.64 [17:57:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.90, 3.64 [17:58:28] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.82, 1.81, 1.63 [17:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.63, 4.00 [18:00:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.20, 3.32 [18:00:22] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.42, 1.68, 1.61 [18:00:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.89, 3.84, 4.04 [18:01:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 4.02, 3.71 [18:03:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.92, 3.22, 3.46 [18:04:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.44, 3.85 [18:04:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.69, 3.79, 3.61 [18:06:07] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.94, 1.85, 1.70 [18:06:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.93, 1.82 [18:06:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.89, 3.97 [18:06:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.61, 3.56 [18:08:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.64, 3.87 [18:09:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.26, 1.83, 1.71 [18:12:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.78, 4.68, 4.19 [18:12:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.68, 4.03, 3.68 [18:13:16] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14598 MB (10% inode=99%); [18:13:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.47, 1.89, 1.80 [18:14:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.72, 3.60 [18:14:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.53, 3.29 [18:15:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.02, 1.94, 1.83 [18:16:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.98, 3.39, 3.27 [18:17:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.81, 1.87, 1.81 [18:19:16] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15591 MB (11% inode=99%); [18:19:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.57, 1.99, 1.86 [18:21:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.40, 1.82, 1.81 [18:22:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 2.92, 3.28 [18:25:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.47, 1.56, 1.70 [18:26:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.76, 3.88, 3.41 [18:28:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.23, 3.84 [18:30:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.44, 3.36 [18:32:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 2.95, 3.17 [18:36:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.74, 3.75 [18:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.82, 3.83 [18:44:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.50, 4.09 [18:44:37] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:45:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.72, 3.26 [18:47:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.34, 3.19 [18:50:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.79, 3.94 [18:55:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.28, 10.33, 9.74 [18:56:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 4.09, 3.97 [18:57:09] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.42, 1.81, 1.54 [18:57:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 3.70, 3.42 [18:58:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.97, 3.95 [18:59:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.56, 1.70, 1.53 [18:59:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.58, 3.40 [19:00:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 4.34, 4.09 [19:01:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.20, 3.28 [19:02:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.84, 3.93 [19:03:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.17, 10.11, 9.96 [19:03:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [19:05:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.00, 1.72, 1.34 [19:05:21] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=criticalrolewiki /home/reception/criticalrole/Critical+Role+Wiki-20220512173218.xml --report 1 --no-updates --username-prefix=wikia:criticalrole (START) [19:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 3.85, 3.88 [19:06:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.12, 11.03, 10.08 [19:07:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.41, 1.58, 1.32 [19:07:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [19:07:10] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.39, 3.17 [19:08:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.60, 1.80, 1.65 [19:10:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.64, 3.29 [19:10:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.50, 1.64, 1.61 [19:10:37] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:10:42] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=criticalrolewiki /home/reception/criticalrole/Critical+Role+Wiki-20220512173218.xml --report 1 --no-updates --username-prefix=wikia:criticalrole (END - exit=0) [19:10:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.44, 10.00, 9.90 [19:10:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=criticalrolewiki --u=Miraheze_maintenance_script --r=Requested at T9216 /home/reception/criticalroledel.txt (START) [19:14:08] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=criticalrolewiki --u=Miraheze_maintenance_script --r=Requested at T9216 /home/reception/criticalroledel.txt (END - exit=256) [19:14:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.57, 3.36 [19:14:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:34] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=criticalrolewiki --u=Miraheze_maintenance_script --r=Requested at T9216 /home/reception/criticalroledel.txt (START) [19:14:35] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=criticalrolewiki --u=Miraheze_maintenance_script --r=Requested at T9216 /home/reception/criticalroledel.txt (END - exit=256) [19:14:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:07] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=criticalrolewiki --u="Miraheze_maintenance_script" --r="Requested at T9216" /home/reception/criticalroledel.txt [19:15:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.75, 3.44 [19:16:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.19, 10.31, 10.03 [19:18:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.32, 3.38, 3.33 [19:20:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.83, 9.91, 9.96 [19:22:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.95, 5.31, 4.17 [19:22:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.73, 4.27, 3.68 [19:23:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [19:23:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.34, 1.76, 1.99 [19:24:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.94, 3.63 [19:25:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.56, 2.22, 2.14 [19:26:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.13, 4.43, 3.85 [19:27:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 10.30, 9.85 [19:29:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 10.96, 10.15 [19:30:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.78, 3.49, 3.81 [19:31:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.55, 10.09 [19:33:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.62, 10.60, 10.03 [19:33:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.96, 3.39, 3.06 [19:34:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 3.99, 3.93 [19:35:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.13, 10.43 [19:35:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.52, 10.66, 10.13 [19:35:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.04, 3.29, 3.06 [19:35:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.75, 1.99 [19:35:44] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:36:38] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [19:37:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.25, 10.16, 10.17 [19:37:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.62, 10.17, 10.01 [19:37:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.60, 2.15, 2.11 [19:39:18] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=criticalrolewiki (START) [19:39:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 10.39, 10.24 [19:47:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.80, 10.12, 10.18 [19:48:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.56, 3.90 [19:48:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [19:48:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.65, 3.98 [19:51:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.01, 1.85, 1.53 [19:53:43] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.53, 3.43, 3.13 [19:53:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.30, 1.59, 1.47 [19:54:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.46, 3.73 [19:55:38] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.09, 3.40, 3.16 [19:56:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 3.66, 3.69 [19:57:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.85, 10.51, 9.39 [19:57:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.32, 2.03, 1.68 [19:58:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.83, 3.76 [19:58:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.86, 3.87 [19:59:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.82, 1.92, 1.68 [20:00:42] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [20:01:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.95, 9.60, 9.31 [20:02:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.05, 2.91, 3.36 [20:03:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.58, 1.69, 1.64 [20:04:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.18, 4.89, 4.19 [20:06:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 4.48, 3.96 [20:06:40] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [20:06:40] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40697169303894 seconds [20:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.91, 3.82 [20:08:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.88, 3.94 [20:10:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.91, 3.82 [20:10:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.94, 3.94 [20:11:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.11, 10.34, 9.59 [20:12:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.63, 3.72 [20:12:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.69, 3.85 [20:13:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.56, 10.00, 9.55 [20:16:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.44, 3.59 [20:18:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.71, 3.77, 3.72 [20:20:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.76, 3.68 [20:20:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.72, 3.73 [20:21:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.25, 10.71, 10.15 [20:21:40] Please did anybody checked gpcommonswiki out? [20:21:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.10, 2.12, 1.77 [20:22:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.37, 10.07, 9.71 [20:22:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 4.00, 3.83 [20:23:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.38, 1.76, 1.67 [20:24:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.47, 3.53 [20:24:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.57, 9.50, 9.54 [20:25:18] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.84, 9.80, 9.96 [20:27:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 5.24, 2.98, 2.12 [20:31:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.12, 10.31, 10.14 [20:31:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.04, 1.88, 1.85 [20:32:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.88, 3.95 [20:34:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 4.04, 3.99 [20:36:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.91, 3.96 [20:37:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.95, 1.38, 1.64 [20:39:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.93, 9.95, 10.11 [20:41:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 1.67, 1.98 [20:42:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.82, 3.98 [20:45:42] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [20:48:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 3.74, 3.86 [20:48:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.83, 3.80, 3.79 [20:49:52] PROBLEM - cp21 Disk Space on cp21 is CRITICAL: DISK CRITICAL - free space: / 2329 MB (5% inode=96%); [20:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.64, 3.80 [20:50:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.44, 3.65 [20:52:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.87, 3.85 [20:52:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.93, 3.97, 3.81 [20:53:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.41, 10.39, 9.57 [20:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [20:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407186269760132 seconds [20:54:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.96, 3.89 [20:54:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.73, 3.75 [20:55:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.67, 9.42, 9.32 [20:55:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.52, 10.28, 9.31 [20:55:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.39, 1.43, 1.65 [20:55:42] alerting : [FIRING:1, RESOLVED:1] (mediawiki) https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki [20:56:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.91, 4.05, 3.86 [20:57:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.59, 9.58, 9.17 [20:57:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.61, 10.42, 10.22 [20:58:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.05, 4.60, 4.12 [20:59:39] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.64, 1.71 [20:59:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.47, 9.76, 10.01 [21:01:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.27, 10.02, 9.58 [21:01:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.40, 1.52, 1.65 [21:03:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.63, 9.87, 9.61 [21:07:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 10.47, 10.16 [21:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.89, 3.90 [21:11:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 11.05, 10.41 [21:13:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 11.19, 10.54 [21:14:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.81, 3.80 [21:16:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.26, 3.18, 3.56 [21:16:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.17, 10.16, 9.40 [21:17:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.18, 9.96, 10.17 [21:18:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.06, 9.80, 9.37 [21:20:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.74, 3.73 [21:20:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.42, 10.34, 9.24 [21:21:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.98, 10.97, 10.52 [21:22:55] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.72, 9.73, 9.15 [21:23:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [21:24:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.65, 3.72 [21:25:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 9.64, 10.11 [21:26:26] PROBLEM - cp20 Disk Space on cp20 is CRITICAL: DISK CRITICAL - free space: / 2344 MB (5% inode=96%); [21:28:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.34, 3.87 [21:29:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.81, 11.08, 10.59 [21:30:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 2.85, 3.33 [21:31:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.26, 9.73, 10.16 [21:32:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.35, 4.00 [21:33:03] [02puppet] 07Universal-Omega opened pull request 03#2588: Reformat grafana-webhook - 13https://github.com/miraheze/puppet/pull/2588 [21:33:04] [url] Page not found · GitHub · GitHub | github.com [21:33:52] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [21:33:52] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:34:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 3.38, 3.64 [21:36:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.00, 3.47 [21:37:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.81, 9.86, 9.99 [21:38:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.62, 3.51 [21:38:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.67, 1.99 [21:38:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.83, 4.24, 4.12 [21:38:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 2.97, 3.40 [21:39:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.15, 9.45, 9.82 [21:39:59] ugochimobi: check out what? [21:40:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.84, 3.98 [21:41:41] PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.creaturathegame.com' expires in 15 day(s) (Thu 09 Jun 2022 21:35:14 GMT +0000). [21:42:05] [02puppet] 07RhinosF1 reviewed pull request 03#2588 commit - 13https://github.com/miraheze/puppet/pull/2588#discussion_r880974055 [21:42:06] [url] Reformat grafana-webhook by Universal-Omega · Pull Request #2588 · miraheze/puppet · GitHub | github.com [21:42:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 3.05, 3.32 [21:42:39] [02puppet] 07RhinosF1 reviewed pull request 03#2588 commit - 13https://github.com/miraheze/puppet/pull/2588#discussion_r880974442 [21:42:39] [url] Reformat grafana-webhook by Universal-Omega · Pull Request #2588 · miraheze/puppet · GitHub | github.com [21:42:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.03, 6.75, 6.21 [21:43:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a09e72cb77d2...6740ff30b707 [21:43:31] [url] Comparing a09e72cb77d2...6740ff30b707 · miraheze/ssl · GitHub | github.com [21:43:32] [02miraheze/ssl] 07MirahezeSSLBot 036740ff3 - Bot: Update SSL cert for wiki.creaturathegame.com [21:43:58] gpcommonswiki isn't rendering images correctly [21:44:21] * ugochimobi maybe a maintainance script would do, but everything there seems laggy [21:44:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.03, 7.12, 6.40 [21:44:59] gpcommonswiki isn't rendering images correctly, maybe a maintenance script would do but everything seems laggy [21:46:05] ugochimobi: is there a task? [21:46:11] Or any error? [21:46:14] Example? [21:46:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.81, 1.92 [21:46:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.64, 3.81 [21:46:34] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.23, 6.52, 6.27 [21:48:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.42, 3.69 [21:49:59] RhinosF1: the wgLogo stopped rendering [21:50:01] https://gpcommons.miraheze.org/wiki/File:Commons_logo_v2.svg [21:50:02] [url] File:Commons logo v2.svg - Gratispaideia Commons | gpcommons.miraheze.org [21:50:35] all images stopped giving thumbs, [21:52:22] ugochimobi: hmm, I see. [21:52:23] PROBLEM - ircwiki.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ircwiki.cf' expires in 15 day(s) (Thu 09 Jun 2022 21:23:54 GMT +0000). [21:52:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.55, 3.73, 3.70 [21:52:34] Has anything changed recently [21:53:16] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14596 MB (10% inode=99%); [21:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [21:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406650066375732 seconds [21:54:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6740ff30b707...c6c259626a86 [21:54:03] [url] Comparing 6740ff30b707...c6c259626a86 · miraheze/ssl · GitHub | github.com [21:54:04] [02miraheze/ssl] 07MirahezeSSLBot 03c6c2596 - Bot: Update SSL cert for ircwiki.cf [21:54:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.10, 4.90, 3.77 [21:54:50] RhinosF1: no, nothing changed at all [21:55:02] ugochimobi: please file a task [21:56:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.89, 4.56, 3.67 [21:58:53] RhinosF1: https://phabricator.miraheze.org/T9271 [21:58:54] [url] ⚓ T9271 Images not rendering correctly on gpcommonswiki | phabricator.miraheze.org [21:58:55] done [21:59:16] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15588 MB (11% inode=99%); [21:59:48] [02puppet] 07Universal-Omega synchronize pull request 03#2588: Reformat grafana-webhook - 13https://github.com/miraheze/puppet/pull/2588 [21:59:49] [url] Page not found · GitHub · GitHub | github.com [22:01:23] [02puppet] 07Universal-Omega reviewed pull request 03#2588 commit - 13https://github.com/miraheze/puppet/pull/2588#discussion_r880985064 [22:01:23] [url] Reformat grafana-webhook by Universal-Omega · Pull Request #2588 · miraheze/puppet · GitHub | github.com [22:01:50] [02puppet] 07Universal-Omega reviewed pull request 03#2588 commit - 13https://github.com/miraheze/puppet/pull/2588#discussion_r880985342 [22:01:51] [url] Reformat grafana-webhook by Universal-Omega · Pull Request #2588 · miraheze/puppet · GitHub | github.com [22:02:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:05:25] PROBLEM - ecole.science - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ecole.science' expires in 15 day(s) (Thu 09 Jun 2022 21:43:27 GMT +0000). [22:05:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.47, 10.81, 9.89 [22:06:39] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [22:07:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c6c259626a86...85dfe6fa2c77 [22:07:14] [url] Comparing c6c259626a86...85dfe6fa2c77 · miraheze/ssl · GitHub | github.com [22:07:16] [02miraheze/ssl] 07MirahezeSSLBot 0385dfe6f - Bot: Update SSL cert for ecole.science [22:07:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.25, 10.86, 10.02 [22:07:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.08, 2.04, 1.45 [22:09:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.27, 10.05, 9.84 [22:09:55] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.51, 1.48, 1.32 [22:10:52] RECOVERY - wiki.creaturathegame.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Mon 22 Aug 2022 20:43:25 GMT +0000. [22:12:55] PROBLEM - wiki.beergeeks.co.il - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.beergeeks.co.il' expires in 15 day(s) (Thu 09 Jun 2022 22:07:00 GMT +0000). [22:14:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.53, 1.81, 1.96 [22:16:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.87, 1.95 [22:19:25] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=criticalrolewiki (END - exit=0) [22:19:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.43, 2.21, 1.51 [22:20:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.80, 1.43 [22:21:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/85dfe6fa2c77...2ef5d511f643 [22:21:06] [url] Comparing 85dfe6fa2c77...2ef5d511f643 · miraheze/ssl · GitHub | github.com [22:21:07] [02miraheze/ssl] 07MirahezeSSLBot 032ef5d51 - Bot: Update SSL cert for wiki.beergeeks.co.il [22:21:50] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.90, 1.69, 1.40 [22:22:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.43, 1.34 [22:23:32] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [22:25:34] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bikinibottomdiscord.org' expires in 15 day(s) (Thu 09 Jun 2022 22:19:55 GMT +0000). [22:26:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.87, 1.57 [22:26:13] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2ef5d511f643...a9c54e6a957d [22:26:13] [url] Comparing 2ef5d511f643...a9c54e6a957d · miraheze/ssl · GitHub | github.com [22:26:14] [02miraheze/ssl] 07MirahezeSSLBot 03a9c54e6 - Bot: Update SSL cert for wiki.bikinibottomdiscord.org [22:26:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.80, 1.95 [22:27:51] PROBLEM - rct.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rct.wiki' expires in 15 day(s) (Thu 09 Jun 2022 22:02:40 GMT +0000). [22:28:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.43, 1.44 [22:33:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a9c54e6a957d...f2fe669ab01c [22:33:37] [url] Comparing a9c54e6a957d...f2fe669ab01c · miraheze/ssl · GitHub | github.com [22:33:38] [02miraheze/ssl] 07MirahezeSSLBot 03f2fe669 - Bot: Update SSL cert for rct.wiki [22:35:16] RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Mon 22 Aug 2022 21:07:09 GMT +0000. [22:36:40] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.re [22:36:40] stname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406361818313599 seconds [22:40:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.69, 3.99 [22:42:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.51, 1.68 [22:42:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.15, 3.80 [22:43:45] PROBLEM - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipedia.yogfront.ooo' expires in 15 day(s) (Thu 09 Jun 2022 22:28:18 GMT +0000). [22:45:18] PROBLEM - wikidiffs.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikidiffs.ga' expires in 15 day(s) (Thu 09 Jun 2022 22:35:31 GMT +0000). [22:46:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.84, 3.82, 3.87 [22:48:06] PROBLEM - www.connorjwat.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.connorjwat.net' expires in 15 day(s) (Thu 09 Jun 2022 22:20:12 GMT +0000). [22:48:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.53, 3.75 [22:48:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.61, 1.94, 1.79 [22:48:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f2fe669ab01c...5acbb925230c [22:48:37] [url] Comparing f2fe669ab01c...5acbb925230c · miraheze/ssl · GitHub | github.com [22:48:38] [02miraheze/ssl] 07MirahezeSSLBot 035acbb92 - Bot: Update SSL cert for wikipedia.yogfront.ooo [22:49:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.93, 9.75, 9.00 [22:50:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.63, 3.75 [22:51:08] RECOVERY - ircwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'ircwiki.cf' will expire on Mon 22 Aug 2022 20:53:57 GMT +0000. [22:51:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.43, 9.49, 8.99 [22:51:47] PROBLEM - wiki.timewisely.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.timewisely.net' expires in 15 day(s) (Thu 09 Jun 2022 22:36:01 GMT +0000). [22:52:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.98, 1.86 [22:52:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.40, 3.83, 3.80 [22:52:49] PROBLEM - threedomwiki.pcast.site - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'threedomwiki.pcast.site' expires in 15 day(s) (Thu 09 Jun 2022 22:32:23 GMT +0000). [22:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [22:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.408218860626221 seconds [22:54:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.57, 3.70 [22:55:13] PROBLEM - www.ambientguitar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ambientguitar.org' expires in 15 day(s) (Thu 09 Jun 2022 22:45:26 GMT +0000). [22:55:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5acbb925230c...cd079aab8efe [22:55:50] [url] Comparing 5acbb925230c...cd079aab8efe · miraheze/ssl · GitHub | github.com [22:55:51] [02miraheze/ssl] 07MirahezeSSLBot 03cd079aa - Bot: Update SSL cert for wikidiffs.ga [22:56:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 4.09, 3.86 [22:56:28] PROBLEM - ambientguitar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ambientguitar.org' expires in 15 day(s) (Thu 09 Jun 2022 22:45:26 GMT +0000). [22:56:42] RECOVERY - rct.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rct.wiki' will expire on Mon 22 Aug 2022 21:33:31 GMT +0000. [22:59:51] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Thu 09 Jun 2022 22:51:24 GMT +0000). [23:00:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.29, 1.53, 1.69 [23:01:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cd079aab8efe...d04f8bb290a5 [23:01:25] PROBLEM - zh.internetpedia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zh.internetpedia.tk' expires in 15 day(s) (Thu 09 Jun 2022 22:39:08 GMT +0000). [23:01:25] [url] Comparing cd079aab8efe...d04f8bb290a5 · miraheze/ssl · GitHub | github.com [23:01:26] [02miraheze/ssl] 07MirahezeSSLBot 03d04f8bb - Bot: Update SSL cert for wiki.timewisely.net [23:01:40] PROBLEM - wikileague.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikileague.cf' expires in 15 day(s) (Thu 09 Jun 2022 22:32:50 GMT +0000). [23:02:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.48, 3.73 [23:02:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d04f8bb290a5...9a26b0378d65 [23:02:39] [url] Comparing d04f8bb290a5...9a26b0378d65 · miraheze/ssl · GitHub | github.com [23:02:40] [02miraheze/ssl] 07MirahezeSSLBot 039a26b03 - Bot: Update SSL cert for threedomwiki.pcast.site [23:02:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.85, 3.86, 3.86 [23:03:23] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 2.56, 1.65, 0.96 [23:03:27] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [23:04:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.78, 4.63, 4.09 [23:04:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.72, 4.59, 4.12 [23:05:21] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 1.45, 1.59, 1.02 [23:05:34] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [23:07:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9a26b0378d65...d812f5af22b4 [23:07:18] [url] Comparing 9a26b0378d65...d812f5af22b4 · miraheze/ssl · GitHub | github.com [23:07:19] [02miraheze/ssl] 07MirahezeSSLBot 03d812f5a - Bot: Update SSL cert for zh.internetpedia.tk [23:07:47] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Thu 09 Jun 2022 22:43:18 GMT +0000). [23:08:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.58, 3.56, 3.77 [23:08:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.73, 3.87 [23:10:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.73, 1.72 [23:10:53] RECOVERY - wiki.beergeeks.co.il - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Mon 22 Aug 2022 21:21:00 GMT +0000. [23:12:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.67, 1.70 [23:13:31] RECOVERY - wikipedia.yogfront.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipedia.yogfront.ooo' will expire on Mon 22 Aug 2022 21:48:31 GMT +0000. [23:13:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d812f5af22b4...c44e06a0b56d [23:13:41] [url] Comparing d812f5af22b4...c44e06a0b56d · miraheze/ssl · GitHub | github.com [23:13:42] [02miraheze/ssl] 07MirahezeSSLBot 03c44e06a - Bot: Update SSL cert for ambientguitar.org [23:15:07] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Thu 09 Jun 2022 22:59:34 GMT +0000). [23:16:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 2.87, 3.36 [23:18:15] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Thu 09 Jun 2022 23:04:27 GMT +0000). [23:18:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.43, 2.07, 1.85 [23:20:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c44e06a0b56d...7d9946f83b32 [23:20:56] [url] Comparing c44e06a0b56d...7d9946f83b32 · miraheze/ssl · GitHub | github.com [23:20:57] [02miraheze/ssl] 07MirahezeSSLBot 037d9946f - Bot: Update SSL cert for www.connorjwat.net [23:21:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.86, 2.08, 1.53 [23:21:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.93, 6.95, 6.01 [23:21:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7d9946f83b32...fc717779081c [23:21:52] [url] Comparing 7d9946f83b32...fc717779081c · miraheze/ssl · GitHub | github.com [23:21:53] [02miraheze/ssl] 07MirahezeSSLBot 03fc71777 - Bot: Update SSL cert for wikileague.cf [23:22:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.79, 4.35, 3.95 [23:23:26] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.41, 6.34, 5.90 [23:23:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - stardewmodding.wiki All nameservers failed to answer the query. [23:24:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.63, 3.51 [23:24:42] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Mon 22 Aug 2022 21:26:08 GMT +0000. [23:25:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.19, 1.79, 1.56 [23:25:21] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Thu 09 Jun 2022 23:18:28 GMT +0000). [23:25:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fc717779081c...e64f25811a65 [23:25:55] [url] Comparing fc717779081c...e64f25811a65 · miraheze/ssl · GitHub | github.com [23:25:56] [02miraheze/ssl] 07MirahezeSSLBot 03e64f258 - Bot: Update SSL cert for wiki.ssij.se [23:26:09] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.83, 3.23, 2.03 [23:26:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.79, 3.60 [23:26:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.95, 1.94 [23:27:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.49, 3.15, 2.03 [23:28:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.62, 2.96, 2.08 [23:28:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 4.12, 3.75 [23:30:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.97, 3.74 [23:30:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e64f25811a65...60f3ab329d41 [23:30:41] [url] Comparing e64f25811a65...60f3ab329d41 · miraheze/ssl · GitHub | github.com [23:30:42] [02miraheze/ssl] 07MirahezeSSLBot 0360f3ab3 - Bot: Update SSL cert for wiki.grottocenter.org [23:30:49] RECOVERY - zh.internetpedia.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.internetpedia.tk' will expire on Mon 22 Aug 2022 22:07:13 GMT +0000. [23:32:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.94, 1.93 [23:32:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/60f3ab329d41...7d6173276dc3 [23:32:39] [url] Comparing 60f3ab329d41...7d6173276dc3 · miraheze/ssl · GitHub | github.com [23:32:41] [02miraheze/ssl] 07MirahezeSSLBot 037d61732 - Bot: Update SSL cert for wiki.tulpa.info [23:32:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7d6173276dc3...f3aae3d26433 [23:32:59] [url] Comparing 7d6173276dc3...f3aae3d26433 · miraheze/ssl · GitHub | github.com [23:33:00] [02miraheze/ssl] 07MirahezeSSLBot 03f3aae3d - Bot: Update SSL cert for encyclopediaofastrobiology.org [23:33:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.69, 9.64, 8.90 [23:34:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.52, 3.94 [23:35:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.91, 8.99, 8.76 [23:36:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.87, 4.64, 3.94 [23:36:12] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Thu 09 Jun 2022 23:18:28 GMT +0000). [23:36:39] PROBLEM - www.stardewmodding.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.stardewmodding.wiki All nameservers failed to answer the query. [23:36:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.85, 4.01 [23:37:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.16, 1.82, 1.91 [23:38:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.99, 1.96 [23:38:41] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Thu 09 Jun 2022 23:31:44 GMT +0000). [23:38:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.69, 3.93 [23:40:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.93, 3.82 [23:40:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.20, 2.04 [23:41:04] [02dns] 07Universal-Omega opened pull request 03#278: Remove old acme challenges for betaheze - 13https://github.com/miraheze/dns/pull/278 [23:41:04] [url] Page not found · GitHub · GitHub | github.com [23:43:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.34, 1.67 [23:44:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.91, 1.96 [23:44:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f3aae3d26433...6a6bf74d67e0 [23:44:38] [url] Comparing f3aae3d26433...6a6bf74d67e0 · miraheze/ssl · GitHub | github.com [23:44:39] [02miraheze/ssl] 07MirahezeSSLBot 036a6bf74 - Bot: Update SSL cert for dariawiki.org [23:44:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.29, 3.61 [23:44:53] RECOVERY - wikidiffs.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'wikidiffs.ga' will expire on Mon 22 Aug 2022 21:55:44 GMT +0000. [23:45:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6a6bf74d67e0...54f28bf7a333 [23:45:05] [url] Comparing 6a6bf74d67e0...54f28bf7a333 · miraheze/ssl · GitHub | github.com [23:45:06] [02miraheze/ssl] 07MirahezeSSLBot 0354f28bf - Bot: Update SSL cert for fc.songcontests.eu [23:45:18] PROBLEM - speleo.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'speleo.wiki' expires in 15 day(s) (Thu 09 Jun 2022 23:28:08 GMT +0000). [23:45:22] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Thu 09 Jun 2022 23:13:25 GMT +0000). [23:45:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/54f28bf7a333...712407770fab [23:45:32] [url] Comparing 54f28bf7a333...712407770fab · miraheze/ssl · GitHub | github.com [23:45:33] [02miraheze/ssl] 07MirahezeSSLBot 037124077 - Bot: Update SSL cert for speleo.wiki [23:46:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.14, 3.75, 3.70 [23:46:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 1.93, 1.95 [23:46:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.37, 3.62 [23:48:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.36, 3.56 [23:49:47] RECOVERY - wiki.timewisely.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.timewisely.net' will expire on Mon 22 Aug 2022 22:01:18 GMT +0000. [23:51:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/712407770fab...4de63250f8df [23:51:50] [url] Comparing 712407770fab...4de63250f8df · miraheze/ssl · GitHub | github.com [23:51:51] [02miraheze/ssl] 07MirahezeSSLBot 034de6325 - Bot: Update SSL cert for adadevelopersacademy.wiki [23:52:05] RECOVERY - threedomwiki.pcast.site - LetsEncrypt on sslhost is OK: OK - Certificate 'threedomwiki.pcast.site' will expire on Mon 22 Aug 2022 22:02:33 GMT +0000. [23:52:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.09, 3.40 [23:52:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.90, 3.08, 3.40 [23:53:36] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolv [23:53:36] me, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406665325164795 seconds [23:53:38] RECOVERY - www.ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Mon 22 Aug 2022 22:13:35 GMT +0000. [23:54:28] RECOVERY - ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Mon 22 Aug 2022 22:13:35 GMT +0000. [23:54:42] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Thu 09 Jun 2022 23:43:40 GMT +0000). [23:54:54] PROBLEM - mediawiki.ecfin.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mediawiki.ecfin.eu' expires in 15 day(s) (Thu 09 Jun 2022 23:52:15 GMT +0000). [23:55:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.06, 10.27, 9.71 [23:57:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.86, 10.05, 9.70 [23:58:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.72, 3.54 [23:58:31] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Mon 22 Aug 2022 22:32:53 GMT +0000. [23:59:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 4.15, 3.78