[00:02:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.06, 8.93 [00:04:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.74, 9.39, 8.81 [00:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:30:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.31, 3.89 [00:33:18] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:35:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 3.69, 3.93 [00:35:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 9.42, 8.91 [00:37:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.60, 3.87 [00:37:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.81, 8.84, 8.76 [00:39:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 4.03, 3.98 [00:41:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.54, 3.80 [00:47:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 4.48, 4.12 [00:55:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.72, 3.96 [01:03:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.13, 4.02 [01:03:13] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:15:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.63, 4.00 [01:16:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.46, 4.00 [01:28:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.39, 3.93 [01:30:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.72, 3.28 [01:31:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.03, 3.40 [01:32:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 3.81, 3.95 [01:34:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.50, 3.82 [01:35:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.33, 3.44 [01:37:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 3.01, 3.31 [01:37:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 4.16, 3.74 [01:40:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.99, 3.92 [01:42:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.87, 3.88 [01:42:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:46:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 4.50, 4.11 [01:47:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 3.94, 3.55 [01:48:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.84, 1.58 [01:49:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.27, 3.36 [01:50:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.53, 1.49 [01:50:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.80, 3.91 [01:51:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.88, 1.39, 1.08 [01:52:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:53:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.58, 1.91, 1.28 [01:54:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.68, 4.14, 3.97 [01:55:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.25, 1.50, 1.20 [01:56:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.81, 3.87 [01:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.95, 3.56 [01:58:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.34, 4.07 [01:59:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.53, 3.43, 3.42 [02:01:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.82, 3.33, 3.39 [02:05:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 3.81, 3.54 [02:09:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.54, 3.49 [02:10:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.18, 3.47, 3.84 [02:11:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.18, 3.36 [02:12:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.89 [02:14:07] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.83, 6.35, 5.62 [02:16:02] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.75, 6.03, 5.58 [02:18:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.81, 3.92 [02:20:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.55, 3.44 [02:22:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.35, 3.38 [02:26:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.82, 3.65, 3.51 [02:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:27:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 4.04, 3.68 [02:28:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 3.96, 3.85 [02:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:34:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [02:34:53] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 9.85, 9.02 [02:37:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.95, 3.89 [02:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.12, 9.74, 9.09 [02:39:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.05, 3.92 [02:41:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.86, 3.86 [02:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.55, 3.93 [02:57:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.90, 3.77, 3.72 [02:59:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.60, 3.35, 3.58 [03:02:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.31, 3.92, 3.91 [03:03:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 2.97, 3.36 [03:10:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.52, 3.76 [03:26:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 3.49, 3.54 [03:28:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.95, 3.73 [03:34:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.72, 2.70, 3.26 [03:34:42] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:34:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.98, 1.45 [03:38:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.65, 1.37, 1.32 [03:43:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.53, 3.36 [03:45:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.18, 3.25 [03:46:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.07, 3.84 [03:50:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 3.51, 3.81 [03:52:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.43, 3.74 [03:54:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.58, 3.74 [03:56:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.52, 3.70 [03:58:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.94, 3.83 [04:00:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.68, 3.76 [04:06:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.61, 3.65 [04:08:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.62, 3.66 [04:12:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 4.18, 3.89 [04:14:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.76, 3.78 [04:22:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 2.98, 3.36 [04:34:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.78, 3.50 [04:36:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.52, 3.45 [04:38:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/11815d750fc0 [04:38:03] [02miraheze/mw-config] 07Universal-Omega 0311815d7 - Fix typo in editextendedconfirmedprotected for lhmnwiki [04:38:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.29, 3.37 [04:38:05] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [04:38:05] [url] Page not found · GitHub · GitHub | github.com [04:38:06] [02mw-config] 07Universal-Omega opened pull request 03#4670: Fix typo in editextendedconfirmedprotected for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/4670 [04:38:07] [url] Page not found · GitHub · GitHub | github.com [04:39:11] miraheze/mw-config - Universal-Omega the build passed. [04:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:42:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.72, 3.61 [04:44:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.91, 3.70 [04:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:44:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.78, 3.18 [04:46:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.60, 3.62 [04:46:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.56, 3.16 [04:48:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.59, 3.21 [04:49:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 3.08, 3.40 [04:50:23] [02mw-config] 07Universal-Omega commented on pull request 03#4669: set wgInterwikiSortingSort to alphabetically on gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4669#issuecomment-1120133712 [04:50:24] [url] set wgInterwikiSortingSort to alphabetically on gratispaideiawiki by ugochimobi · Pull Request #4669 · miraheze/mw-config · GitHub | github.com [04:50:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.58, 3.24 [04:52:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 3.09, 3.10 [05:30:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.12, 2.80 [05:31:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.60, 3.17 [05:32:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.76, 2.72 [05:33:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 3.21, 3.08 [05:46:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.52, 3.23 [05:50:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.87, 3.43 [05:52:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.34, 3.28 [06:04:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.44, 3.30 [06:04:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.12, 3.44 [06:06:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.65, 3.35 [06:08:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 2.83, 3.09 [06:08:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.25, 3.23 [06:32:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.70, 3.10 [06:32:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.45, 2.93 [06:34:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.69, 2.96, 2.90 [06:34:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.84, 2.93, 2.80 [06:39:00] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [06:39:00] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:44:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.52, 3.06 [06:46:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.64, 3.15 [06:46:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.21, 3.39 [06:48:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.45, 3.15, 3.03 [06:48:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.66, 3.28 [06:50:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 3.99, 3.44 [06:52:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.34, 3.46, 3.32 [06:54:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 2.98, 3.17 [07:00:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.90, 3.41 [07:02:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.80, 3.43 [07:10:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 2.98, 3.23 [07:23:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.94, 5.75, 5.02 [07:25:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.53, 5.69, 5.09 [07:29:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.24, 2.99 [07:33:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 2.79, 2.88 [07:43:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.97, 4.22, 3.47 [07:47:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.55, 3.68, 3.46 [07:51:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.34, 3.37 [07:55:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.75, 3.51 [07:59:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.42, 4.24, 3.73 [08:07:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.89, 3.84 [08:08:44] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:13:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.31, 3.53 [08:17:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.28, 3.47 [08:19:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.53, 3.53 [08:21:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.84, 3.43, 3.48 [08:23:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 2.99, 3.31 [08:23:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.76, 2.13, 1.34 [08:25:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.68, 1.61, 1.24 [08:30:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.48, 3.50 [08:32:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 3.19, 3.40 [08:41:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.56, 3.43 [08:43:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.04, 3.25 [09:00:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.38, 3.24 [09:02:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.43, 2.69, 3.01 [09:05:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.19, 2.55 [09:07:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 2.75, 2.47 [09:31:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.22, 3.08 [09:33:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.58, 3.90, 3.33 [09:35:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.62, 3.32, 3.19 [09:40:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:40:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:45:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.37, 2.51 [10:45:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.67, 4.55, 3.00 [10:46:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.43, 4.41, 3.16 [10:47:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.42, 2.65 [10:47:17] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.27, 1.90, 1.36 [10:49:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.94, 2.93 [10:49:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.34, 2.16, 1.52 [10:53:17] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.88, 1.62, 1.46 [10:57:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.94, 1.63 [10:59:17] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 1.71, 1.58 [11:01:17] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.59, 2.13, 1.76 [11:03:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.93, 3.78 [11:03:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.97, 1.86, 1.69 [11:05:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.07, 3.84 [11:07:17] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.04, 1.64, 1.65 [11:09:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.67, 3.74 [11:10:09] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:16:54] [02RottenLinks] 07kewilk commented on pull request 03#59: 3 fixes and a new config variable - 13https://github.com/miraheze/RottenLinks/pull/59#issuecomment-1120190561 [11:16:55] [url] 3 fixes and a new config variable by kewilk · Pull Request #59 · miraheze/RottenLinks · GitHub | github.com [11:17:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.36, 3.66, 3.59 [11:19:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.68, 3.61 [11:21:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.84, 4.17, 3.78 [11:23:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.77, 3.69 [11:27:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.68, 3.64 [11:29:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.66, 3.37, 3.54 [11:31:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.47, 3.96 [11:39:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 3.35, 3.61 [11:39:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.10, 3.16, 3.39 [11:41:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.41, 3.61 [11:41:59] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:43:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.68, 3.68 [11:45:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.56, 3.66 [11:49:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.97, 3.81 [11:53:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.60, 3.42 [11:57:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.32, 3.48, 3.75 [11:57:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.06, 3.25 [12:01:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.69, 3.76 [12:04:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.50, 3.67 [12:06:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.21, 3.94, 3.79 [12:08:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 3.91, 3.48 [12:08:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.76, 3.74 [12:10:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.42, 3.34 [12:10:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 3.82, 3.73 [12:11:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:12:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 3.01, 3.21 [12:16:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.90, 3.80 [12:19:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.47, 3.29 [12:23:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.62, 2.92, 3.14 [12:24:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.76, 3.32 [12:28:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.60, 3.52 [12:29:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.66, 3.36 [12:30:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.34, 3.43 [12:31:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 3.11, 3.19 [12:32:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 3.03, 3.30 [12:50:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:50:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.18, 3.08, 4.00 [12:51:57] PROBLEM - petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petrawiki.org could not be found [12:55:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:58:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.44, 3.75 [12:59:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.11, 10.31, 9.34 [12:59:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.89, 3.49 [13:00:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.32, 3.68 [13:00:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:01:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.26, 9.91, 9.31 [13:02:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.71, 4.45, 4.06 [13:06:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.91, 3.92 [13:06:17] PROBLEM - files.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for files.petrawiki.org could not be found [13:08:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.85, 4.88, 4.26 [13:15:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.03, 4.29, 3.54 [13:18:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.17, 10.30, 9.95 [13:19:09] PROBLEM - www.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.petrawiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:21:04] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address petrawiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:21:57] PROBLEM - www.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.petrawiki.org could not be found [13:24:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.28, 9.95, 9.91 [13:25:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.88, 3.71 [13:30:32] PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address files.petrawiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:31:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.19, 4.62, 4.04 [13:32:21] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address hr.petrawiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:35:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.72, 3.80 [13:35:08] PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address translate.petrawiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:36:35] PROBLEM - en.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.petrawiki.org could not be found [13:39:38] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f01b7eb789d6 [13:39:39] [url] Page not found · GitHub · GitHub | github.com [13:39:40] [02miraheze/mw-config] 07RhinosF1 03f01b7eb - Licenses: Add GFDL [13:39:41] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://github.com/miraheze/mw-config [13:39:42] [url] Page not found · GitHub · GitHub | github.com [13:39:43] [02mw-config] 07RhinosF1 opened pull request 03#4671: Licenses: Add GFDL - 13https://github.com/miraheze/mw-config/pull/4671 [13:39:43] [url] Page not found · GitHub · GitHub | github.com [13:40:11] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hr.petrawiki.org could not be found [13:40:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:40:39] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f01b7eb789d6...310da74b4291 [13:40:40] [url] Comparing f01b7eb789d6...310da74b4291 · miraheze/mw-config · GitHub | github.com [13:40:40] [02miraheze/mw-config] 07RhinosF1 03310da74 - Update ManageWikiSettings.php [13:40:42] [02mw-config] 07RhinosF1 synchronize pull request 03#4671: Licenses: Add GFDL - 13https://github.com/miraheze/mw-config/pull/4671 [13:40:43] ... [13:40:48] miraheze/mw-config - RhinosF1 the build passed. [13:41:29] [02mw-config] 07RhinosF1 closed pull request 03#4670: Fix typo in editextendedconfirmedprotected for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/4670 [13:41:30] [url] Page not found · GitHub · GitHub | github.com [13:41:31] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5ee639bf6519...52be0a6993c9 [13:41:31] [url] Comparing 5ee639bf6519...52be0a6993c9 · miraheze/mw-config · GitHub | github.com [13:41:32] [02miraheze/mw-config] 07Universal-Omega 0352be0a6 - Fix typo in editextendedconfirmedprotected for lhmnwiki (#4670) [13:41:34] [02miraheze/mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-1 [13:41:35] [02mw-config] 07RhinosF1 deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [13:41:36] [url] Page not found · GitHub · GitHub | github.com [13:41:52] miraheze/mw-config - RhinosF1 the build passed. [13:42:32] miraheze/mw-config - RhinosF1 the build passed. [13:44:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.38, 9.58, 8.67 [13:45:00] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for translate.petrawiki.org could not be found [13:47:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:48:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.50, 9.14, 8.71 [13:48:40] PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address en.petrawiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:49:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.05, 3.40 [13:50:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 9.73, 9.04 [13:52:35] !log [@mwtask111] starting deploy of {'config': True} to all [13:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:52:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.46, 9.41, 8.99 [13:52:45] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [13:52:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.64, 2.87, 3.64 [13:56:31] !log [@test101] starting deploy of {'config': True} to all [13:56:33] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 2s [13:56:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:50] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolv [13:56:50] ve(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407292127609253 seconds [13:59:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.41, 3.28 [14:01:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.87, 3.13, 3.19 [14:09:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.95, 3.55 [14:09:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 4.52, 3.99 [14:15:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.33, 3.40 [14:17:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:19:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.89, 3.99 [14:20:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.49, 9.83, 8.82 [14:22:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.36, 9.14, 8.68 [14:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:23:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.27, 4.23, 4.03 [14:25:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 4.00, 3.98 [14:26:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.56, 1.81, 1.33 [14:26:49] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:27:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.95, 3.62 [14:28:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:28:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.03, 1.50, 1.28 [14:29:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.64, 3.54 [14:29:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.06, 4.23, 4.06 [14:31:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.57, 10.56, 9.80 [14:34:18] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.78, 1.94, 1.49 [14:35:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.09, 3.70, 3.58 [14:36:17] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.76, 1.47 [14:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:36:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.32, 10.32 [14:37:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 3.14, 3.39 [14:38:16] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.49, 1.56, 1.42 [14:38:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 11.36, 10.46 [14:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:46:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.86, 9.91, 10.15 [14:47:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [14:47:15] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:49:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.31, 3.28 [14:51:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.91, 3.06, 3.19 [14:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:56:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.91, 9.72, 8.87 [14:57:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.13, 10.46, 10.29 [14:58:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 10.48, 9.54 [14:58:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.10, 8.68, 8.60 [14:58:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.53, 3.37 [15:00:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.84, 3.49 [15:01:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.03, 9.99, 10.17 [15:02:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.93, 3.48, 3.40 [15:05:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.60, 4.00 [15:06:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 2.95, 3.21 [15:07:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.72, 3.98 [15:08:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.86, 10.17, 9.94 [15:09:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:12:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 10.46, 10.16 [15:12:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.49, 10.15, 9.22 [15:14:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.19, 9.37, 9.05 [15:16:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.38, 3.27 [15:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:17:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:18:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.64, 10.00, 10.02 [15:18:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.02, 3.18, 3.20 [15:19:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 10.43, 10.08 [15:19:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.56, 3.89 [15:25:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.90, 3.87 [15:26:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.98, 10.41, 10.14 [15:28:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.08, 9.91, 9.99 [15:29:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.86, 3.90 [15:30:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.70, 3.37 [15:30:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 11.40, 10.72 [15:31:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 4.17, 4.01 [15:32:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 11.09, 10.69 [15:37:11] PROBLEM - hi.famepedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hi.famepedia.org' expires in 15 day(s) (Mon 23 May 2022 15:24:15 GMT +0000). [15:38:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.83, 10.12, 9.97 [15:38:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.47, 3.40, 3.39 [15:40:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.65, 9.55, 9.78 [15:44:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ba8db6a274b...1f3a819e77bb [15:44:22] [url] Comparing 0ba8db6a274b...1f3a819e77bb · miraheze/ssl · GitHub | github.com [15:44:23] [02miraheze/ssl] 07MirahezeSSLBot 031f3a819 - Bot: Update SSL cert for hi.famepedia.org [15:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.74, 9.16, 9.93 [15:46:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.41, 3.38 [15:48:10] [02RottenLinks] 07Universal-Omega reviewed pull request 03#59 commit - 13https://github.com/miraheze/RottenLinks/pull/59#discussion_r867366198 [15:48:10] [url] 3 fixes and a new config variable by kewilk · Pull Request #59 · miraheze/RottenLinks · GitHub | github.com [15:49:01] [02RottenLinks] 07RhinosF1 synchronize pull request 03#59: 3 fixes and a new config variable - 13https://github.com/miraheze/RottenLinks/pull/59 [15:49:01] [url] Page not found · GitHub · GitHub | github.com [15:49:35] [02RottenLinks] 07Universal-Omega reviewed pull request 03#59 commit - 13https://github.com/miraheze/RottenLinks/pull/59#discussion_r867366288 [15:49:36] [url] 3 fixes and a new config variable by kewilk · Pull Request #59 · miraheze/RottenLinks · GitHub | github.com [15:49:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.82, 10.26, 9.38 [15:50:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.68, 4.28, 3.67 [15:51:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.83, 9.86, 9.33 [15:52:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.67, 3.51 [15:53:09] miraheze/RottenLinks - RhinosF1 the build passed. [15:53:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.57, 3.92 [15:54:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.19, 3.35 [15:55:24] [02RottenLinks] 07Universal-Omega synchronize pull request 03#59: 3 fixes and a new config variable - 13https://github.com/miraheze/RottenLinks/pull/59 [15:55:24] [url] Page not found · GitHub · GitHub | github.com [15:55:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.78, 3.95 [15:56:30] [02RottenLinks] 07Universal-Omega closed pull request 03#59: 3 fixes and a new config variable - 13https://github.com/miraheze/RottenLinks/pull/59 [15:56:31] [url] Page not found · GitHub · GitHub | github.com [15:56:32] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/RottenLinks/compare/a5c783cfc70c...0a1ab39cdb06 [15:56:32] [url] Comparing a5c783cfc70c...0a1ab39cdb06 · miraheze/RottenLinks · GitHub | github.com [15:56:33] [02miraheze/RottenLinks] 07kewilk 030a1ab39 - 3 fixes and a new config variable (#59) [15:57:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:57:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.74, 3.91 [16:00:06] miraheze/RottenLinks - Universal-Omega the build passed. [16:02:05] miraheze/RottenLinks - Universal-Omega the build passed. [16:03:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.82, 1.24 [16:03:33] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/MatomoAnalytics/commit/7191428d0f5f [16:03:34] [02miraheze/MatomoAnalytics] 07Universal-Omega 037191428 - Use markdown format for CHANGELOG [16:03:36] [02MatomoAnalytics] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MatomoAnalytics [16:03:36] [url] Page not found · GitHub · GitHub | github.com [16:03:37] [02MatomoAnalytics] 07Universal-Omega opened pull request 03#77: Use markdown format for CHANGELOG - 13https://github.com/miraheze/MatomoAnalytics/pull/77 [16:03:38] [url] Page not found · GitHub · GitHub | github.com [16:05:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.67, 1.38, 1.15 [16:05:12] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#77: Use markdown format for CHANGELOG - 13https://github.com/miraheze/MatomoAnalytics/pull/77 [16:05:13] ... [16:05:14] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/MatomoAnalytics/compare/79374aa88e00...5710ffe9f17e [16:05:14] [url] Comparing 79374aa88e00...5710ffe9f17e · miraheze/MatomoAnalytics · GitHub | github.com [16:05:15] [02miraheze/MatomoAnalytics] 07Universal-Omega 035710ffe - Use markdown format for CHANGELOG (#77) [16:05:17] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [16:05:18] [02MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MatomoAnalytics [16:05:19] [url] Page not found · GitHub · GitHub | github.com [16:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 9.88, 9.51 [16:07:05] RECOVERY - hi.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hi.famepedia.org' will expire on Fri 05 Aug 2022 14:44:14 GMT +0000. [16:08:11] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-1/±0] 13https://github.com/miraheze/IncidentReporting/commit/e5bb96c96ceb [16:08:12] [02miraheze/IncidentReporting] 07Universal-Omega 03e5bb96c - Use markdown format for CHANGELOG [16:08:14] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/IncidentReporting [16:08:15] [url] Page not found · GitHub · GitHub | github.com [16:08:16] [02IncidentReporting] 07Universal-Omega opened pull request 03#42: Use markdown format for CHANGELOG - 13https://github.com/miraheze/IncidentReporting/pull/42 [16:08:16] ... [16:08:36] miraheze/MatomoAnalytics - Universal-Omega the build passed. [16:09:50] [02IncidentReporting] 07Universal-Omega closed pull request 03#42: Use markdown format for CHANGELOG - 13https://github.com/miraheze/IncidentReporting/pull/42 [16:09:51] [url] Page not found · GitHub · GitHub | github.com [16:09:51] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/IncidentReporting/compare/2a4ae16b7b80...e07936868a70 [16:09:52] [url] Comparing 2a4ae16b7b80...e07936868a70 · miraheze/IncidentReporting · GitHub | github.com [16:09:53] [02miraheze/IncidentReporting] 07Universal-Omega 03e079368 - Use markdown format for CHANGELOG (#42) [16:09:54] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [16:09:56] [02IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/IncidentReporting [16:09:57] [url] Page not found · GitHub · GitHub | github.com [16:10:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.62, 9.85 [16:10:55] miraheze/MatomoAnalytics - Universal-Omega the build passed. [16:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:12:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.39, 10.52, 9.91 [16:12:55] miraheze/IncidentReporting - Universal-Omega the build passed. [16:15:20] miraheze/IncidentReporting - Universal-Omega the build passed. [16:16:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:16:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.94, 9.70, 9.71 [16:19:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [16:19:53] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:20:46] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [16:20:46] [url] Page not found · GitHub · GitHub | github.com [16:20:47] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/5b751270dfea...f94fd78d53a8 [16:20:48] [url] Comparing 5b751270dfea...f94fd78d53a8 · miraheze/IncidentReporting · GitHub | github.com [16:20:49] [02miraheze/IncidentReporting] 07Universal-Omega 03f94fd78 - Update ext.incidentreporting.oouiform.ooui.js [16:21:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.01, 3.35 [16:24:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.33, 3.14, 3.92 [16:24:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.77, 10.29, 9.92 [16:25:26] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/f94fd78d53a8...a23e1e937870 [16:25:26] [url] Comparing f94fd78d53a8...a23e1e937870 · miraheze/IncidentReporting · GitHub | github.com [16:25:27] miraheze/IncidentReporting - Universal-Omega the build passed. [16:25:27] [02miraheze/IncidentReporting] 07Universal-Omega 03a23e1e9 - Update extension.json [16:25:29] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [16:25:29] [url] Page not found · GitHub · GitHub | github.com [16:26:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.64, 4.01 [16:26:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.14, 9.81, 9.78 [16:26:50] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [16:26:50] [url] Page not found · GitHub · GitHub | github.com [16:26:51] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/a23e1e937870...7ab083e14a1e [16:26:52] [url] Comparing a23e1e937870...7ab083e14a1e · miraheze/IncidentReporting · GitHub | github.com [16:26:53] [02miraheze/IncidentReporting] 07Universal-Omega 037ab083e - Update SpecialIncidentReports.php [16:31:59] miraheze/IncidentReporting - Universal-Omega the build passed. [16:32:59] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/7ab083e14a1e...cb3a6b470ea2 [16:33:00] [url] Comparing 7ab083e14a1e...cb3a6b470ea2 · miraheze/IncidentReporting · GitHub | github.com [16:33:01] [02miraheze/IncidentReporting] 07Universal-Omega 03cb3a6b4 - Update ext.incidentreporting.oouiform.ooui.js [16:33:02] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [16:33:03] [url] Page not found · GitHub · GitHub | github.com [16:33:43] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6613d5dabf18...00b8136d7d4d [16:33:44] [url] Comparing 6613d5dabf18...00b8136d7d4d · miraheze/ImportDump · GitHub | github.com [16:33:44] [02miraheze/ImportDump] 07Universal-Omega 0300b8136 - Update ext.importdump.oouiform.ooui.js [16:33:46] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:33:47] [url] Page not found · GitHub · GitHub | github.com [16:35:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:36:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.50, 3.12 [16:36:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.61, 3.36 [16:36:42] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/cb3a6b470ea2...3a114e8352ab [16:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 10.40, 10.11 [16:36:42] [url] Comparing cb3a6b470ea2...3a114e8352ab · miraheze/IncidentReporting · GitHub | github.com [16:36:43] [02miraheze/IncidentReporting] 07Universal-Omega 033a114e8 - Update SpecialIncidentReports.php [16:36:45] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [16:36:45] [url] Page not found · GitHub · GitHub | github.com [16:37:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.50, 3.16 [16:38:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.83, 4.23, 3.61 [16:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.44, 9.94, 9.97 [16:39:08] miraheze/IncidentReporting - Universal-Omega the build passed. [16:39:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.99, 3.38 [16:42:14] miraheze/IncidentReporting - Universal-Omega the build passed. [16:42:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 10.30, 10.12 [16:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.32, 9.68, 9.92 [16:45:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:45:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.44, 3.61, 3.48 [16:45:57] miraheze/ImportDump - Universal-Omega the build passed. [16:47:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 3.30, 3.38 [16:48:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.90, 3.89 [16:49:47] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:50:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.71, 4.57, 4.11 [16:56:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.50, 9.36, 8.43 [16:58:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.44, 9.59, 8.64 [16:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.64, 3.89 [16:58:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 10.55, 10.07 [17:00:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 4.07, 3.61 [17:00:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:02:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.38, 3.41 [17:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.41, 11.81, 9.75 [17:04:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.99, 3.01, 3.28 [17:04:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.26, 11.01, 9.70 [17:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 11.69, 10.71 [17:06:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.88, 10.75, 9.39 [17:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 11.19, 10.67 [17:06:45] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:24] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:47] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.45, 11.62, 10.29 [17:10:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.69, 11.72, 10.07 [17:11:25] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.69, 10.11, 8.45 [17:11:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.88, 10.53, 9.40 [17:11:56] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:10] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.012 second response time [17:12:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 2.88, 3.31 [17:12:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.44, 11.62, 11.01 [17:13:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.66, 10.20, 8.73 [17:13:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.88, 10.02, 8.63 [17:13:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.23, 11.17, 9.77 [17:13:45] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:55] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.42, 6.42, 5.49 [17:14:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 11.62, 11.09 [17:15:01] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.51, 9.36, 8.62 [17:15:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.82, 11.18, 9.96 [17:17:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.03, 11.64, 10.27 [17:18:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.49, 11.91, 11.13 [17:18:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.65, 11.99, 10.93 [17:20:15] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.181 second response time [17:20:28] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 9.177 second response time [17:20:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.70, 12.37, 11.54 [17:20:46] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.361 second response time [17:21:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.77, 11.59, 10.60 [17:21:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.61, 6.20, 5.90 [17:21:56] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.501 second response time [17:22:03] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 5.902 second response time [17:22:19] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.204 second response time [17:22:43] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.938 second response time [17:23:15] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.019 second response time [17:23:16] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.246 second response time [17:23:16] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.728 second response time [17:24:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.20, 1.75, 1.21 [17:25:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.39, 10.08, 10.19 [17:25:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:26:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.46, 11.10, 11.33 [17:26:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.31, 1.24, 1.09 [17:27:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.59, 3.42 [17:28:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.36, 8.89, 10.03 [17:30:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.69, 9.03, 9.92 [17:33:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.04, 3.29 [17:35:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.06, 8.81, 9.98 [17:40:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:49:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.64, 3.31 [17:53:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.79, 3.46 [17:57:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.80, 3.52 [17:59:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.32, 3.38 [18:00:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.44, 3.16 [18:01:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.80, 2.95 [18:05:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.44, 3.19 [18:07:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.54, 3.47 [18:07:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 10.18, 9.46 [18:07:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.18, 3.25, 3.14 [18:09:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 10.82, 9.76 [18:10:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.72, 10.14, 8.77 [18:11:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.76, 3.54 [18:11:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.94, 4.02, 3.49 [18:12:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.85, 10.03, 8.93 [18:12:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.42, 9.78, 8.74 [18:12:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.29, 10.38, 9.85 [18:13:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.93, 3.54 [18:14:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.46, 8.84, 8.52 [18:15:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.39, 3.39 [18:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:17:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.85, 3.67 [18:20:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.60, 9.30, 9.65 [18:22:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.49, 3.44 [18:23:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.67, 3.63 [18:26:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 4.15, 3.71 [18:27:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.46, 3.58 [18:30:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.17, 10.48, 10.12 [18:32:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.15, 9.20, 9.69 [18:33:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.76, 4.17, 3.80 [18:34:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.30, 11.01, 9.76 [18:35:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.59, 3.63 [18:35:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.84, 3.81 [18:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 10.12, 9.96 [18:36:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.58, 10.72, 9.81 [18:39:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.53, 3.64 [18:40:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 11.15, 10.40 [18:41:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.47, 3.18, 3.51 [18:42:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 10.73, 10.35 [18:43:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.52, 3.55 [18:44:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.52, 10.04, 9.95 [18:44:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.66, 10.31, 9.19 [18:45:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.04, 3.38 [18:46:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.86, 9.85, 9.16 [18:47:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.81, 3.66 [18:50:17] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:39] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 11.25, 10.73 [18:50:46] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 3.440 second response time [18:52:34] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.124 second response time [18:52:35] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.333 second response time [18:52:40] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.400 second response time [18:52:47] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.027 second response time [18:54:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.42, 11.23, 10.91 [18:55:39] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.13, 1.47, 1.19 [18:56:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 11.88, 11.18 [18:57:39] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.64, 1.18, 1.12 [18:58:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 11.39, 11.10 [18:59:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.00, 3.33 [19:01:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.99, 9.19, 8.02 [19:03:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.21, 8.58, 7.93 [19:08:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.64, 9.03, 10.08 [19:09:53] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14113 MB (10% inode=99%); [19:09:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.73, 5.49, 3.62 [19:11:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.61, 5.35, 3.80 [19:11:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.67, 3.46 [19:15:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.33, 3.35 [19:15:57] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.53, 5.04, 4.03 [19:19:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.88, 3.58 [19:21:16] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [19:25:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.23, 4.39, 3.60 [19:25:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.21, 10.34, 9.95 [19:27:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.95, 3.53 [19:27:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 5.12, 6.06, 5.08 [19:29:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.76, 3.76 [19:31:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.44, 3.37, 3.40 [19:31:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.70, 10.11, 10.01 [19:33:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.02, 4.12, 3.88 [19:36:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 10.73, 10.24 [19:37:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.41, 3.44 [19:37:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.95, 3.37, 3.67 [19:38:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.33, 10.19, 10.10 [19:39:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.40, 3.26, 3.37 [19:39:53] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7520 MB (5% inode=99%); [19:47:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.37, 4.22, 3.58 [19:47:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.76, 4.47, 3.94 [19:47:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.57, 5.08, 5.82 [19:51:11] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [19:51:11] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:53:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.19, 6.83, 6.17 [19:53:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.17, 7.02, 6.32 [19:55:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.89, 3.76 [19:55:14] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.03, 6.45, 6.12 [19:55:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.16, 5.77, 5.95 [19:57:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.90, 4.07, 3.85 [19:57:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.30, 7.20, 6.45 [19:59:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.68, 3.74 [20:01:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.03, 3.85 [20:02:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.87, 10.44, 10.15 [20:03:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.86, 3.82 [20:09:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.67, 1.43 [20:11:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.57, 1.43 [20:11:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.29, 9.52, 10.00 [20:13:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.75, 9.97, 9.53 [20:13:53] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15651 MB (11% inode=99%); [20:13:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.65, 4.50, 5.67 [20:15:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.24, 9.14, 9.30 [20:15:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.96, 12.44, 8.70 [20:19:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.17, 2.88, 3.35 [20:19:53] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13483 MB (10% inode=99%); [20:21:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.64, 1.96 [20:21:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.66, 3.98 [20:25:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.81, 10.46, 10.04 [20:25:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.59, 1.68 [20:26:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.60, 9.82, 9.87 [20:31:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 3.86, 3.79 [20:33:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.44, 3.64 [20:39:53] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7926 MB (5% inode=99%); [20:41:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.05, 3.36 [20:45:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 3.66, 3.51 [20:46:03] CosmicAlpha: I doubt if RhinosF1 remembered to run rebuildLinks.php for gratispaideiawiki yesterday, can you please do that? [20:47:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.44, 3.44 [20:48:21] .in 30mins ugochimobi's request [20:48:21] RhinosF1: Okay, will remind at 2022-05-07 - 22:18:21BST [20:49:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.29, 3.40 [20:50:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.08, 3.14, 3.82 [20:53:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.75, 3.58 [20:55:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.10, 4.29, 3.78 [20:56:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.43, 4.27, 4.06 [20:56:51] oh, you around, I thought you nipped out [20:57:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.55, 3.54, 3.56 [20:57:48] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.56, 3.33 [20:58:52] ugochimobi: I'm on paid irc now [20:58:57] So never disconnect [20:59:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.45, 4.70, 5.81 [21:01:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.87, 3.48 [21:02:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:03:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.69, 3.45 [21:03:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.35, 4.28, 3.83 [21:04:58] RhinosF1: oh, okay. [21:05:34] I wanted to do that too, but I don't just want it, lol [21:05:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.98, 3.77 [21:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:09:21] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.21, 3.29, 3.32 [21:09:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 4.15, 3.87 [21:09:57] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.84, 4.13, 5.00 [21:11:53] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 17569 MB (13% inode=99%); [21:13:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.92, 3.86 [21:15:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.20, 4.64, 4.11 [21:17:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 1.63, 1.29 [21:17:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.73, 6.69, 5.72 [21:18:23] RhinosF1: ugochimobi's request [21:19:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.35, 1.23 [21:19:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.82, 9.73, 8.93 [21:19:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.13, 7.23, 6.04 [21:20:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.94, 10.61, 9.46 [21:21:52] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.53, 9.68, 9.01 [21:21:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.60, 6.88, 6.05 [21:22:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.82, 9.68 [21:23:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.00, 6.72, 6.10 [21:24:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.22, 9.89, 9.48 [21:25:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.81, 3.90 [21:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:27:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.22, 6.97, 6.38 [21:29:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.29, 5.90, 6.05 [21:31:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.64, 3.69 [21:32:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:33:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.50, 4.28, 3.54 [21:33:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.72, 3.72 [21:33:58] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:11] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:35] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.13, 11.44, 10.22 [21:34:54] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:56] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:57] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:07] Huh [21:35:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 16.00, 12.92, 10.64 [21:35:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.78, 3.44 [21:35:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.30, 13.10, 9.76 [21:35:35] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:41] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:47] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:56] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:59] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 3.319 second response time [21:36:08] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.86, 12.40, 9.54 [21:36:11] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.861 second response time [21:36:32] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.019 second response time [21:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.24, 10.25, 9.94 [21:36:43] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:54] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.791 second response time [21:36:57] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.314 second response time [21:37:30] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.027 second response time [21:37:46] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 9.203 second response time [21:37:52] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 7.341 second response time [21:37:53] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.333 second response time [21:38:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.47, 10.84, 9.33 [21:38:11] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 6.618 second response time [21:38:44] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 2.126 second response time [21:38:59] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.916 second response time [21:39:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.21, 3.30 [21:39:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 11.23, 9.91 [21:40:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 9.66, 9.08 [21:41:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.08, 9.90, 9.58 [21:41:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 3.73, 3.62 [21:42:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:42:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.39, 8.72, 8.28 [21:42:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 11.38, 11.13 [21:44:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.87, 8.80, 8.37 [21:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.24, 10.10, 10.05 [21:46:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 11.24, 11.09 [21:47:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:47:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.71, 3.71 [21:48:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.19, 10.66, 10.89 [21:50:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.08, 11.29, 11.09 [21:50:50] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [21:53:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 10.50, 10.23 [21:54:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.14, 10.50, 10.88 [21:55:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.89, 3.81, 3.30 [21:55:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.54, 1.93, 1.35 [21:57:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.60, 1.31 [21:57:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.91, 10.08, 10.12 [21:59:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.58, 3.32 [21:59:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.09, 3.38 [22:01:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.99, 3.39, 3.28 [22:02:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:03:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.67, 3.56 [22:05:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.15, 3.37 [22:06:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.94, 2.99, 3.95 [22:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 10.28, 10.08 [22:09:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:10:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.26, 10.50 [22:12:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.61, 11.07, 10.54 [22:15:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 3.80, 2.44 [22:15:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.62, 3.44 [22:16:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.04, 9.94, 10.16 [22:18:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.16, 2.93, 3.34 [22:19:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.91, 3.66 [22:20:45] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:23:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.75, 3.62 [22:24:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.87, 9.55, 10.02 [22:24:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.73, 10.57, 10.32 [22:25:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.83, 3.67 [22:26:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.03, 10.18, 9.54 [22:28:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.30, 10.09, 9.61 [22:29:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.91, 1.76, 1.97 [22:31:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.48, 9.80, 8.74 [22:31:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.81, 3.28 [22:32:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.46, 11.56, 10.65 [22:32:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.60, 12.05, 10.52 [22:32:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.07, 10.79, 9.46 [22:32:40] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:54] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 1.86, 1.95 [22:33:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 10.12, 8.69 [22:33:38] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:41] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:49] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.74, 6.83, 5.47 [22:34:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.63, 11.05, 9.72 [22:34:50] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.99, 1.52, 1.81 [22:35:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.04, 10.37, 8.96 [22:35:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.63, 6.58, 5.54 [22:36:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.17, 11.14, 10.72 [22:36:34] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:36] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:58] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 9.158 second response time [22:37:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 24.61, 9.54, 4.61 [22:37:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.33, 9.91, 8.96 [22:37:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.62, 9.14, 8.94 [22:37:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.23, 4.19, 3.63 [22:37:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.38, 4.14, 3.34 [22:38:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.71, 4.64, 3.68 [22:38:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.42, 11.12, 10.70 [22:38:52] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:39:57] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.80, 11.20, 10.76 [22:40:09] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:09] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:15] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:31] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.28, 10.14, 9.73 [22:40:42] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:51] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:55] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:11] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:13] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:13] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:24] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.40, 10.22, 9.41 [22:41:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:35] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.90, 3.43 [22:41:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.85, 7.34, 6.10 [22:42:02] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.893 second response time [22:42:09] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.786 second response time [22:42:13] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.658 second response time [22:42:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 11.25, 10.79 [22:42:47] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.030 second response time [22:43:00] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [22:43:09] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.015 second response time [22:43:10] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 1.089 second response time [22:43:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 5 backends are down. mw101 mw102 mw112 mw121 mw122 [22:43:28] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.018 second response time [22:43:35] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.316 second response time [22:43:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 4.34, 3.66 [22:43:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.51, 6.35, 5.88 [22:44:03] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.489 second response time [22:44:07] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.912 second response time [22:44:15] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.167 second response time [22:44:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.42, 9.93, 10.33 [22:44:48] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 4.535 second response time [22:44:54] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.386 second response time [22:44:56] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 2.813 second response time [22:44:56] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.331 second response time [22:44:59] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.325 second response time [22:44:59] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.955 second response time [22:45:00] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 2.649 second response time [22:45:05] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.095 second response time [22:45:21] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.825 second response time [22:45:23] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 1.691 second response time [22:45:25] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.766 second response time [22:45:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.33, 10.72, 9.60 [22:45:51] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.274 second response time [22:46:19] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 87% [22:46:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.15, 9.35, 10.15 [22:47:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.46, 9.74, 9.40 [22:47:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.00, 8.86, 9.20 [22:48:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.49, 11.11, 11.13 [22:48:19] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 3% [22:49:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.28, 2.98, 1.96 [22:49:19] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [22:49:47] currently experiencing 502s and 3s [22:50:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.49, 9.49, 10.08 [22:50:37] ugochimobi: we're aware [22:50:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.28, 10.38, 10.38 [22:50:47] I see [22:52:58] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:07] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:53:16] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:30] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:33] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:56] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.010 second response time [22:54:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.73, 1.65, 1.70 [22:55:40] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 9.104 second response time [22:56:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.445 second response time [22:56:58] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.081 second response time [22:57:16] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.306 second response time [22:57:39] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 3.867 second response time [22:58:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.42, 9.00, 10.09 [22:59:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.11, 1.19, 1.85 [23:00:05] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:06] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.09, 9.73, 10.20 [23:03:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.38, 1.82, 1.91 [23:04:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 10.13, 10.22 [23:04:54] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 287s [23:05:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.04, 1.43, 1.76 [23:05:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:20] !log killed l10n-update due to incident [23:05:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 11.03, 10.64 [23:07:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.33, 1.68 [23:08:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.71, 9.95, 10.16 [23:21:06] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [23:23:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.62, 7.03, 6.27 [23:23:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [23:23:28] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:25:10] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.02, 7.24, 6.43 [23:27:05] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.62, 6.89, 6.40 [23:28:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.86, 9.95, 10.20 [23:29:01] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.05, 6.54, 6.33 [23:34:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.96, 10.73, 10.41 [23:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.26, 9.65, 10.11 [23:41:08] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:20] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:21] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:25] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:03] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.329 second response time [23:43:19] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.332 second response time [23:43:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.613 second response time [23:43:23] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.124 second response time [23:45:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.44, 9.76, 8.79 [23:47:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.63, 9.14, 8.67 [23:50:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.64, 10.36, 9.98 [23:52:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.78, 10.19, 9.97 [23:53:22] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [23:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki