[00:01:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.90, 4.03, 3.87 [00:01:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.71, 3.52 [00:02:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.49, 2.07, 1.78 [00:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:08:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [00:08:28] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:14:54] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.67, 1.21 [00:16:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.13, 1.49, 1.19 [00:23:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:23:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.78, 3.98 [00:23:43] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.91, 2.54, 1.58 [00:24:45] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.92, 2.29, 1.60 [00:25:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 4.10, 4.06 [00:25:43] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.40, 1.78, 1.41 [00:26:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.04, 1.85, 1.53 [00:27:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.33, 1.33, 1.29 [00:28:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.91, 1.66, 1.50 [00:28:55] [02puppet] 07Universal-Omega edited pull request 03#2537: Rewrite PHP configuration - 13https://github.com/miraheze/puppet/pull/2537 [00:28:55] [url] Page not found · GitHub · GitHub | github.com [00:31:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.66, 3.92 [00:34:14] [02puppet] 07Universal-Omega synchronize pull request 03#2537: Rewrite PHP configuration - 13https://github.com/miraheze/puppet/pull/2537 [00:34:14] [url] Page not found · GitHub · GitHub | github.com [00:37:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.44, 4.20, 4.04 [00:37:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:39:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.95, 3.97 [00:41:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.61, 3.94 [00:45:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.99, 4.18, 4.02 [00:47:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.18, 4.13, 3.93 [00:51:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.97, 3.95 [00:53:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 4.22, 4.03 [00:57:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.36, 9.39 [01:05:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.28, 10.04, 9.70 [01:09:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.65, 3.95 [01:09:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.77, 3.97 [01:11:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 4.44, 4.21 [01:11:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.67, 4.62, 4.25 [01:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:25:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.65, 3.96 [01:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:38:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [01:38:28] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:41:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.77, 4.19, 3.86 [01:45:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.65, 3.75 [01:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:47:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.82, 3.80 [01:49:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.48, 3.96 [01:49:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.38, 3.64 [01:53:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.87, 2.73, 3.32 [01:59:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.55, 4.02, 3.90 [02:01:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.26, 3.30, 3.65 [02:04:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.70, 4.00 [02:06:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.43, 3.79, 4.01 [02:07:27] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [02:08:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.42, 3.84 [02:15:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.42, 3.87, 3.68 [02:16:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.32, 4.50, 4.10 [02:20:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.18, 9.72, 8.21 [02:22:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.88, 9.22, 8.23 [02:22:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 10.35, 8.93 [02:22:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.47, 3.30 [02:23:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.16, 3.38, 3.60 [02:24:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.58, 9.70, 8.87 [02:24:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.79, 3.92 [02:24:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.26, 3.24 [02:26:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.44, 3.84, 3.89 [02:28:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.69, 3.83 [02:30:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 10.31, 9.40 [02:30:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.96, 3.92 [02:32:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.75, 3.37 [02:34:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.66, 9.58, 9.34 [02:34:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.65, 3.37 [02:34:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.88, 3.93 [02:39:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.84, 3.48 [02:41:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.48, 3.39 [02:43:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.16, 3.29 [02:45:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.63, 2.79, 3.30 [02:50:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.25, 3.64, 3.66 [02:52:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.34, 3.55 [02:55:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.71, 3.48 [02:56:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.52, 4.47, 3.96 [03:00:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.48, 3.33 [03:03:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.92, 3.76 [03:04:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.40, 3.38 [03:13:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.52, 3.62 [03:17:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.73, 3.70 [03:22:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.43, 2.90, 3.70 [03:26:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.38, 3.31 [03:27:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 2.72, 3.27 [03:38:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [03:38:28] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:47:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.46, 4.38, 3.57 [03:47:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.70, 3.21 [03:49:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.95, 3.51 [03:49:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.47, 3.19 [03:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:51:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.84, 3.37 [03:52:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.33, 3.34 [03:53:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 3.32, 3.22 [03:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:57:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.60, 3.33 [03:59:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.08, 3.18 [04:37:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.40, 3.25 [04:39:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.21, 3.28, 3.23 [04:47:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.41, 3.30 [04:49:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.34, 3.23, 3.23 [05:03:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.83, 3.43 [05:04:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.30, 8.76, 7.90 [05:04:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 10.32, 9.45 [05:05:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.96, 3.34 [05:05:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.85, 3.48 [05:06:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.56, 8.55, 7.95 [05:06:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.12, 10.07, 9.49 [05:07:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.56, 3.27 [05:08:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 3.77, 3.04 [05:09:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.97, 4.49, 3.66 [05:09:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.15, 4.79, 3.91 [05:12:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.46, 3.11 [05:13:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.46, 3.09, 3.30 [05:13:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.81, 3.73 [05:14:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.11, 3.02 [05:21:27] [02ssl] 07ajhalili2006 commented on pull request 03#514: Add redirects for wiki.rtapp.tk and wiki.recaptime.tk as per T9210 - 13https://github.com/miraheze/ssl/pull/514#issuecomment-1126644194 [05:21:28] [url] Add redirects for wiki.rtapp.tk and wiki.recaptime.tk as per T9210 by ajhalili2006 · Pull Request #514 · miraheze/ssl · GitHub | github.com [05:22:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.65, 4.00, 3.71 [05:23:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.15, 3.11 [05:23:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.51, 3.10 [05:25:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.90, 3.10, 3.10 [05:25:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.29, 3.27, 3.06 [05:26:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.68, 3.67 [05:42:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.05, 3.37 [05:48:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.70, 3.53 [05:50:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.54, 3.49 [05:52:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.26, 3.29, 3.39 [05:55:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.10, 2.81 [05:56:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.73, 3.20 [05:56:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.63, 3.51 [05:57:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.32, 3.05, 2.82 [05:58:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.62, 3.22 [06:00:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 2.97, 3.29 [06:02:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 2.91, 3.03 [06:06:28] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:09:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.03, 2.93 [06:11:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 2.85, 2.87 [06:16:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.51, 3.86, 3.23 [06:17:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.60, 3.19 [06:18:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.53, 3.19 [06:19:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.32, 3.13 [06:22:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 3.12, 3.13 [06:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:34:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.32, 10.60, 9.17 [06:36:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 10.21, 9.21 [06:37:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.19, 2.97 [06:38:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.70, 3.34 [06:38:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.75, 9.11, 8.94 [06:39:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.59, 2.53, 2.75 [06:40:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 3.07, 3.15 [06:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:14:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 4.61, 3.77 [07:15:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.77, 4.34, 3.56 [07:18:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.26, 3.95, 3.74 [07:19:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.57, 3.45 [07:21:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.16, 3.31 [07:26:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 2.81, 3.26 [07:42:39] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [07:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:53:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 10.40, 9.11 [07:55:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.70, 10.00, 9.13 [08:01:18] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jeremyboyd.org' expires in 15 day(s) (Mon 30 May 2022 07:57:26 GMT +0000). [08:03:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:04:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b64071a91f5c...297ec9b00034 [08:04:49] [url] Comparing b64071a91f5c...297ec9b00034 · miraheze/ssl · GitHub | github.com [08:04:50] [02miraheze/ssl] 07MirahezeSSLBot 03297ec9b - Bot: Update SSL cert for wiki.jeremyboyd.org [08:10:39] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:30:36] RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Fri 12 Aug 2022 07:04:44 GMT +0000. [08:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:33:07] PROBLEM - lakehub.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lakehub.ga' expires in 15 day(s) (Mon 30 May 2022 08:17:02 GMT +0000). [08:36:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/297ec9b00034...56eb388c3d83 [08:36:43] [url] Comparing 297ec9b00034...56eb388c3d83 · miraheze/ssl · GitHub | github.com [08:36:44] [02miraheze/ssl] 07MirahezeSSLBot 0356eb388 - Bot: Update SSL cert for lakehub.ga [08:41:20] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanon.polandballwiki.com' expires in 15 day(s) (Mon 30 May 2022 08:25:11 GMT +0000). [08:41:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/56eb388c3d83...4e0d75eb8031 [08:41:40] [url] Comparing 56eb388c3d83...4e0d75eb8031 · miraheze/ssl · GitHub | github.com [08:41:41] [02miraheze/ssl] 07MirahezeSSLBot 034e0d75e - Bot: Update SSL cert for fanon.polandballwiki.com [08:47:25] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sarovia.graalmilitary.com' expires in 15 day(s) (Mon 30 May 2022 08:21:49 GMT +0000). [08:48:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4e0d75eb8031...ae32fa02c016 [08:48:34] [url] Comparing 4e0d75eb8031...ae32fa02c016 · miraheze/ssl · GitHub | github.com [08:48:35] [02miraheze/ssl] 07MirahezeSSLBot 03ae32fa0 - Bot: Update SSL cert for sarovia.graalmilitary.com [09:02:45] RECOVERY - lakehub.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'lakehub.ga' will expire on Fri 12 Aug 2022 07:36:35 GMT +0000. [09:10:09] RECOVERY - fanon.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanon.polandballwiki.com' will expire on Fri 12 Aug 2022 07:41:34 GMT +0000. [09:10:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:10:15] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:12:00] [02mw-config] 07RhinosF1 closed pull request 03#4693: PageImages: make MediaWiki:Pageimages-denylist work everywhere - 13https://github.com/miraheze/mw-config/pull/4693 [09:12:00] [url] Page not found · GitHub · GitHub | github.com [09:12:01] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/mw-config [09:12:02] [url] Page not found · GitHub · GitHub | github.com [09:12:03] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9c0da0770a66...ca9868001dc6 [09:12:04] [url] Comparing 9c0da0770a66...ca9868001dc6 · miraheze/mw-config · GitHub | github.com [09:12:04] [02miraheze/mw-config] 07RhinosF1 03ca98680 - PageImages: make MediaWiki:Pageimages-denylist work everywhere (#4693) [09:12:06] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [09:13:06] miraheze/mw-config - RhinosF1 the build passed. [09:16:21] RECOVERY - sarovia.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sarovia.graalmilitary.com' will expire on Fri 12 Aug 2022 07:48:29 GMT +0000. [09:22:58] !log [@mwtask111] starting deploy of {'config': True} to all [09:23:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:07] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [09:23:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:26] !log [@test101] starting deploy of {'config': True} to all [09:24:27] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [09:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:34:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.41, 2.92 [09:36:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.79, 2.76, 2.74 [09:39:13] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:48:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.73, 3.15 [09:50:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.24, 3.04 [10:32:04] [discord] if anyone from here is on board soon/now, I'd like to enable SMW for a wiki - naturally requiring a script to be run once I do [10:45:43] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.86, 1.96, 1.37 [10:49:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.32, 2.83 [10:49:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.79, 1.37, 1.26 [10:50:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.92, 3.22 [10:51:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 3.17, 2.85 [10:51:20] [discord] @raidarr still needed? [10:51:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.91, 1.82, 1.43 [10:53:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.39, 1.65, 1.41 [10:56:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.18, 3.11 [10:59:24] [discord] aye [10:59:41] [discord] @raidarr can do then [10:59:48] [discord] which wiki? [10:59:49] [discord] got it in a tab, should be done in a moment [11:00:10] [discord] `constantnoblewiki` as requested on the SN [11:00:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.51, 3.26 [11:00:24] !log [www-data@mw101] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=65280) [11:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:00:32] !log [www-data@mw102] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=65280) [11:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:00:49] [discord] did not set up the extension just yet, that was the first step right? [11:00:54] [discord] @raidarr ye [11:01:05] [discord] will note when I have it checked [11:01:30] [discord] alright, it's in [11:01:41] !log [www-data@mw101] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=65280) [11:01:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:21] !log [www-data@mw102] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=0) [11:02:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.64, 3.32 [11:02:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:35] !log [www-data@mw112] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=0) [11:02:39] !log [www-data@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=256) [11:02:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:54] !log [www-data@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=0) [11:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:52] !log [www-data@mw101] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=0) [11:03:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:04:13] !log [www-data@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=0) [11:04:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:04:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.51, 3.32 [11:04:29] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=65280) [11:04:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:04:45] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=constantnoblewiki (END - exit=0) [11:04:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:04:54] [discord] @raidarr done [11:10:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.96, 3.40, 3.34 [11:12:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.94, 3.13 [11:14:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.83, 3.18 [11:16:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.73, 3.06, 2.97 [11:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.38, 3.41, 3.05 [11:32:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.77, 3.46 [11:34:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.94, 2.92 [11:34:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.88, 3.52 [11:35:45] [discord] nice, though it does look like there's a key error accessing the wiki [11:38:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.56, 3.52 [11:44:40] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.08, 3.33 [11:52:36] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [12:02:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.31, 2.91 [12:04:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.10, 3.18, 2.91 [12:13:06] I'm [12:13:08] Mobile now [12:13:13] Not sure what that is [12:13:18] I just got told to run the script [12:13:22] CosmicAlpha: ^ [12:17:30] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.77, 1.65, 1.34 [12:19:30] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.16, 1.43, 1.29 [12:20:36] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [12:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:28:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:36:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.74, 9.81, 8.55 [12:38:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [12:38:28] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:38:58] [02puppet] 07paladox reviewed pull request 03#2537 commit - 13https://github.com/miraheze/puppet/pull/2537#discussion_r872975346 [12:38:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.11, 9.11, 8.46 [12:38:59] [url] Rewrite PHP configuration by Universal-Omega · Pull Request #2537 · miraheze/puppet · GitHub | github.com [12:39:12] [02puppet] 07paladox reviewed pull request 03#2537 commit - 13https://github.com/miraheze/puppet/pull/2537#discussion_r872975346 [12:39:12] [url] Rewrite PHP configuration by Universal-Omega · Pull Request #2537 · miraheze/puppet · GitHub | github.com [12:41:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:44:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.69, 3.11 [12:46:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.60, 3.16 [12:46:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.96, 4.08, 3.60 [12:47:43] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.63, 2.85, 1.93 [12:47:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/15f19021fe99 [12:47:50] [02miraheze/puppet] 07paladox 0315f1902 - php: install yaml [12:47:52] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [12:47:52] [url] Page not found · GitHub · GitHub | github.com [12:47:53] [02puppet] 07paladox opened pull request 03#2538: php: install yaml - 13https://github.com/miraheze/puppet/pull/2538 [12:47:54] [url] Page not found · GitHub · GitHub | github.com [12:48:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.47, 3.43 [12:50:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 3.89, 3.57 [12:51:43] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.00, 1.76, 1.68 [12:51:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.25, 3.18 [12:52:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.58, 3.49 [12:53:43] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.81, 1.43, 1.57 [12:56:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:56:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.96, 3.66 [12:56:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1d68a0e58747...665814f44b4b [12:56:54] [url] Comparing 1d68a0e58747...665814f44b4b · miraheze/puppet · GitHub | github.com [12:56:55] [02miraheze/puppet] 07paladox 03665814f - php: only install php-fpm prometheus exporter if fpm is enabled [12:57:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.75, 4.49, 3.70 [12:59:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.11, 4.43, 3.43 [13:01:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.75, 3.60 [13:03:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.76, 4.62, 3.92 [13:05:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:05:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.55, 3.37 [13:05:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 4.00, 3.76 [13:06:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.96, 3.89 [13:07:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.18, 3.24 [13:07:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.89, 3.73 [13:07:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:09:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.03, 3.20, 3.50 [13:10:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:10:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/76b176763ace [13:10:54] [02miraheze/puppet] 07paladox 0376b1767 - redis: log to redis-server.log as well as syslog [13:10:55] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [13:10:56] [url] Page not found · GitHub · GitHub | github.com [13:10:57] [02puppet] 07paladox opened pull request 03#2539: redis: log to redis-server.log as well as syslog - 13https://github.com/miraheze/puppet/pull/2539 [13:10:57] [url] Page not found · GitHub · GitHub | github.com [13:11:03] [02puppet] 07paladox closed pull request 03#2539: redis: log to redis-server.log as well as syslog - 13https://github.com/miraheze/puppet/pull/2539 [13:11:03] [url] Page not found · GitHub · GitHub | github.com [13:11:04] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [13:11:05] ... [13:11:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [13:11:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/665814f44b4b...fef79b5d77f2 [13:11:08] [url] Comparing 665814f44b4b...fef79b5d77f2 · miraheze/puppet · GitHub | github.com [13:11:09] [02miraheze/puppet] 07paladox 03fef79b5 - redis: log to redis-server.log as well as syslog (#2539) [13:11:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.00, 3.38 [13:12:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.28, 3.95, 3.84 [13:15:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.56, 3.53 [13:17:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.69, 3.59 [13:21:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.43, 3.29 [13:22:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.83, 3.93 [13:23:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.22, 3.17, 3.20 [13:25:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.45, 2.90, 3.27 [13:25:41] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[redis-server] [13:26:12] PROBLEM - test101 JobRunner Service on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:26:27] PROBLEM - test101 JobChron Service on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [13:26:27] PROBLEM - test101 Redis Process on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redis-server' [13:27:53] paladox: redis crashed ^ [13:32:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 4.16, 3.97 [13:34:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.91, 3.91 [13:34:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 0.96, 1.15, 1.95 [13:36:23] looking [13:36:56] it's running according to systemctl logs [13:37:13] https://www.irccloud.com/pastebin/EY9Yx7s3/ [13:38:06] https://www.irccloud.com/pastebin/vPmec3FX/ [13:38:16] ohh test101 [13:38:37] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.85, 0.98, 1.69 [13:40:27] RECOVERY - test101 Redis Process on test101 is OK: PROCS OK: 1 process with args 'redis-server' [13:40:39] !log test101: mkdir /var/log/redis/ [13:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:40:48] !log test101: chown -R redis:adm /var/log/redis/ [13:40:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:41:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.68, 3.49 [13:41:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.83, 3.78, 3.26 [13:42:12] RECOVERY - test101 JobRunner Service on test101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:42:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.14, 4.36, 3.96 [13:42:27] RECOVERY - test101 JobChron Service on test101 is OK: PROCS OK: 1 process with args 'redisJobChronService' [13:43:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.85, 3.36 [13:43:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.74, 3.54 [13:45:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.94, 3.45 [13:45:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.62, 3.97, 3.64 [13:46:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.85, 3.85 [13:47:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.37, 3.45 [13:49:28]