[00:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:17:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.47, 10.20, 9.42 [00:19:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.87, 9.62, 9.30 [00:19:37] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 4772s [00:19:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.78, 9.32, 8.35 [00:21:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.78, 9.29, 8.45 [00:22:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.48, 2.65, 1.42 [00:23:14] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.51, 1.83, 1.27 [00:24:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.40, 9.73, 9.53 [00:25:14] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.156 second response time [00:26:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.48, 1.20 [00:26:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.51, 9.89, 9.63 [00:43:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.34, 3.95 [00:45:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.45, 3.90 [00:47:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:47:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.43, 3.83 [00:51:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.60, 3.31, 3.90 [00:51:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.32, 6.58, 5.53 [00:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:53:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.73, 6.68, 5.70 [01:01:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 2.90, 3.40 [01:02:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.31, 9.79, 9.57 [01:03:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.46, 4.00 [01:07:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 2.62, 3.25 [01:11:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.46, 3.91, 3.62 [01:11:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 3.75, 3.81 [01:13:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.51, 3.42 [01:13:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.83, 3.64 [01:15:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 3.14, 3.30 [01:15:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.59, 3.02, 3.36 [01:17:27] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.19, 9.50, 9.86 [01:17:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.87, 3.86 [01:19:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.82, 3.83 [01:21:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.72, 3.80 [01:24:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 10.33, 10.17 [01:25:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.41, 9.99, 10.09 [01:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:35:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.38, 3.00, 3.38 [01:45:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.69, 3.50 [01:46:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.52, 1.27 [01:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.20, 1.19 [01:48:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.67, 4.65, 3.72 [01:51:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.64, 3.59 [01:52:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.14, 9.81, 9.20 [01:52:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.98, 3.69 [01:54:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.53, 9.93, 9.10 [01:54:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.99, 9.62, 9.22 [01:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:56:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.15, 9.36, 9.00 [01:56:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.75, 3.64 [01:57:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 3.04, 3.35 [02:01:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.89, 3.65 [02:02:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.79, 9.78 [02:04:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.88, 9.93, 9.56 [02:05:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.59, 3.60 [02:05:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.56, 3.12 [02:06:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.33, 1.53, 1.94 [02:07:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.30, 3.09 [02:11:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:11:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.69, 3.61 [02:14:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.24, 3.90 [02:15:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.80, 3.69 [02:17:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 4.05, 3.80 [02:18:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.73, 1.77 [02:18:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.76, 3.93 [02:18:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.36, 0.99 [02:20:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.13, 4.45, 3.59 [02:20:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.29, 1.32, 1.02 [02:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.86, 1.83 [02:26:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.35, 3.89, 3.71 [02:29:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:30:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.57, 1.59, 1.69 [02:30:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.36, 3.89 [02:30:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [02:31:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.43, 3.91 [02:31:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.03, 3.39 [02:33:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.38, 6.54, 5.70 [02:35:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.24, 5.98, 5.60 [02:38:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 4.01, 3.98 [02:39:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.24, 3.61 [02:41:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.26, 3.56 [02:42:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.28, 3.64, 3.89 [02:43:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 2.72, 3.32 [02:44:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:50:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.63, 10.69, 9.95 [02:50:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 1.94, 1.68 [02:50:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 2.74, 3.38 [02:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:52:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.92, 1.71 [02:53:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.83, 3.56 [02:55:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.46, 9.75, 9.80 [02:56:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.93, 10.27, 8.93 [02:58:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.69, 9.64, 8.84 [02:59:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 4.25, 3.45 [02:59:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.20, 4.67, 3.90 [03:00:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.88, 1.74 [03:00:33] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:01:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.87, 3.40, 3.24 [03:01:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.92, 3.72 [03:01:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.78, 3.73 [03:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.33, 11.25, 9.34 [03:02:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.76, 1.71 [03:02:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 10.24, 9.30 [03:03:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.93, 3.74 [03:04:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.73, 10.70, 9.38 [03:04:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.97, 9.45, 9.14 [03:05:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.24, 3.87 [03:06:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.20, 11.13, 9.69 [03:07:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 4.07, 3.56 [03:07:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.68, 1.23 [03:07:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.99, 3.84 [03:08:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.53, 10.19, 9.53 [03:08:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.59, 1.67, 1.70 [03:09:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.42, 1.48, 1.21 [03:09:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 4.17, 3.92 [03:13:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.97, 3.67 [03:16:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:17:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.11, 3.39 [03:17:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.79, 3.96 [03:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.41, 2.04, 1.81 [03:21:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.08, 5.46, 4.50 [03:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.93, 1.81 [03:23:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.04, 5.00, 4.08 [03:28:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.56, 1.68 [03:29:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:32:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.38, 9.84, 9.52 [03:34:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.29, 9.30, 9.37 [03:38:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 10.21, 9.93 [03:40:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.71, 1.67 [03:40:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 9.76, 9.25 [03:42:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.67, 1.69, 1.67 [03:44:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.34, 9.88, 9.50 [03:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.55, 9.59, 9.78 [03:48:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.79, 2.09, 1.82 [03:49:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.11, 3.11, 3.92 [03:51:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.58, 1.84, 1.48 [03:53:50] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.84, 1.57, 1.43 [03:55:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.59, 3.82 [03:56:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.88, 1.87 [03:57:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.72, 3.84 [03:59:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.22, 4.01, 3.92 [04:01:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.96, 3.92 [04:04:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:05:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.42, 3.97 [04:06:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.52, 1.69 [04:06:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.09, 9.77, 9.46 [04:09:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 2.78, 3.36 [04:10:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.08, 9.90, 9.59 [04:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.33, 3.13, 3.43 [04:15:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.24, 2.94, 3.33 [04:15:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 4.01, 3.93 [04:16:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:17:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.58, 3.78 [04:19:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.09, 3.94 [04:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.84, 1.72 [04:21:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.39, 3.69 [04:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.84, 1.74 [04:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.22, 3.96 [04:26:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.59, 1.66 [04:29:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.55, 3.57 [04:30:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 3.91, 3.95 [04:31:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.38, 3.50 [04:33:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:33:11] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:33:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.59, 3.55 [04:35:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.53, 3.54 [04:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 10.42, 9.61 [04:40:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.61, 3.90 [04:43:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.51, 3.90, 3.64 [04:44:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.79, 3.89 [04:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.99, 10.20, 9.90 [04:47:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.82, 3.70 [04:48:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.64, 3.43, 3.74 [04:49:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:50:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.62, 2.17, 1.79 [04:50:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 3.97, 3.90 [04:51:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.10, 10.37, 10.01 [04:52:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.51, 4.57, 3.63 [04:53:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 10.13, 9.98 [04:53:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 4.13, 3.85 [04:54:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.85, 1.75 [04:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:55:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.64, 3.71 [04:57:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.98, 3.83 [04:58:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.60, 1.67 [04:58:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.82, 3.61 [04:59:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.67, 3.73 [05:03:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 4.02, 3.87 [05:04:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.82, 4.12, 3.74 [05:05:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.84, 3.83 [05:05:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.44, 3.53 [05:07:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.06, 3.91 [05:09:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.01, 3.32 [05:13:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.86, 3.88 [05:14:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.26, 3.36, 3.92 [05:14:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:20:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.65, 10.15, 9.38 [05:20:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.18, 2.19, 1.79 [05:21:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.60, 3.24 [05:22:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.84, 9.35, 9.19 [05:22:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 2.53, 3.33 [05:26:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.96, 1.85 [05:33:00] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:33:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.88, 4.33, 3.78 [05:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:35:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.71, 3.62 [05:36:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.56, 1.68 [05:39:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.97, 10.31, 9.10 [05:40:08] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.22, 10.72, 9.51 [05:40:35] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:54] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.29, 10.65, 9.63 [05:41:19] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.42, 10.03 [05:41:29] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:43] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:45] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.86, 11.31, 9.62 [05:41:50] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:42:44] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:42:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:42:51] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.31, 3.08, 3.34 [05:43:22] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.035 second response time [05:43:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.49, 11.16, 10.10 [05:43:44] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 4.139 second response time [05:43:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.01, 11.10, 9.74 [05:43:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.82, 6.73, 5.49 [05:45:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 11.45, 9.30 [05:46:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.39, 11.96, 10.60 [05:47:16] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:20] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.73, 10.73, 9.29 [05:48:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.73, 12.31, 10.89 [05:49:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 11.76, 10.65 [05:49:07] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.89, 1.74 [05:49:14] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.712 second response time [05:49:18] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.854 second response time [05:49:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.88, 10.04, 9.21 [05:49:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.28, 11.96, 10.49 [05:49:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.65, 8.12, 6.47 [05:50:05] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.068 second response time [05:50:16] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 7.715 second response time [05:50:39] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.739 second response time [05:51:23] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.665 second response time [05:53:05] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.96, 1.82 [05:53:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.26, 11.63, 10.03 [05:53:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.22, 7.41, 6.57 [05:54:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 11.82, 11.07 [05:55:04] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 10.73, 9.89 [05:55:47] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.58, 6.73, 6.42 [05:56:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.92, 3.43 [05:56:33] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.46, 3.14 [05:57:09] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.254 second response time [05:57:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.58, 10.13, 9.78 [05:57:43] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 0.298 second response time [05:57:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.22, 11.75, 11.05 [05:58:11] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.399 second response time [05:58:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.99, 11.63, 11.31 [05:58:31] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 2.192 second response time [05:59:18] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 7.816 second response time [05:59:20] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 7.119 second response time [05:59:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.87, 6.84, 6.58 [05:59:56] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.845 second response time [06:00:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.65, 11.63, 11.38 [06:00:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 9.586 second response time [06:00:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.30, 3.14 [06:00:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.90, 11.95, 11.34 [06:01:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.10, 1.43, 1.63 [06:01:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.318 second response time [06:01:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.91, 6.52, 6.49 [06:01:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.36, 2.88 [06:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.07, 12.06, 11.56 [06:02:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 11.49, 11.27 [06:04:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.27, 11.29, 11.36 [06:04:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.30, 8.68, 10.18 [06:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.71, 12.19, 11.54 [06:05:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.44, 8.91, 9.93 [06:05:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.55, 4.17, 3.28 [06:06:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.91, 3.45 [06:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.28, 11.80, 11.47 [06:08:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.32, 3.66 [06:08:58] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.68, 2.16, 1.87 [06:10:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.94, 8.79, 10.14 [06:13:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:24:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 10.70, 10.58 [06:26:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.14, 10.48, 10.52 [06:28:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 11.33, 10.84 [06:30:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.10, 11.16, 10.84 [06:31:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.73, 3.97 [06:32:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.59, 10.47, 9.81 [06:33:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 4.17, 4.12 [06:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.25, 9.60, 9.55 [06:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.47, 10.99 [06:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.17, 11.34, 11.01 [06:40:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 11.35, 11.05 [06:41:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.42, 3.63, 3.98 [06:42:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 11.04, 10.98 [06:44:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.37, 3.98 [06:45:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.02, 3.55, 3.81 [06:46:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.02, 4.14 [06:47:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.53, 3.78 [06:48:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.17, 2.98, 3.78 [06:48:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.57, 3.96 [06:48:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:50:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.40, 3.83 [06:52:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.23, 3.71 [06:52:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.29, 4.53, 4.20 [06:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:53:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.78, 3.61, 3.64 [06:54:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 3.99, 3.95 [06:55:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.63, 3.65 [06:56:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.67, 3.85 [06:57:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.75, 3.68 [06:58:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 3.84, 3.88 [06:59:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.75, 3.53, 3.62 [07:00:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.63, 9.45, 10.16 [07:02:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.53, 3.77 [07:03:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:04:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.65, 3.91 [07:09:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.12, 3.40 [07:10:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 3.93, 3.88 [07:12:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.41, 3.70 [07:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 10.40, 10.00 [07:14:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.05, 3.40 [07:17:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.81, 10.03, 9.98 [07:20:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.47, 4.25, 3.84 [07:22:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.10, 10.77, 10.17 [07:24:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.49, 9.93, 9.94 [07:28:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.73, 3.86 [07:32:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.70, 3.79 [07:34:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.99, 4.08, 3.65 [07:36:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.33, 3.50, 3.49 [07:36:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.87, 3.85 [07:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:38:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.86, 3.62 [07:38:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 4.13, 3.95 [07:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:42:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.13, 10.28, 9.94 [07:44:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.16, 9.89, 9.84 [07:46:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.67, 3.75 [07:46:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.78, 3.93 [07:50:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.24, 3.51 [07:52:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.18, 3.44 [07:54:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.74, 3.24 [07:54:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 10.45, 10.02 [07:56:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.17, 10.01, 9.89 [07:58:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 2.93, 3.35 [08:00:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.52, 3.39 [08:00:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 10.74, 10.21 [08:02:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 3.09, 3.24 [08:02:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:06:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 4.10, 3.63 [08:06:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.41, 9.93, 10.06 [08:08:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.74, 3.57 [08:14:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.05, 3.10, 3.39 [08:26:02] PROBLEM - pokemonwiki.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pokemonwiki.info' expires in 15 day(s) (Tue 24 May 2022 08:04:59 GMT +0000). [08:27:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1f3a819e77bb...8eb5baa62dc5 [08:27:07] [url] Comparing 1f3a819e77bb...8eb5baa62dc5 · miraheze/ssl · GitHub | github.com [08:27:08] [02miraheze/ssl] 07MirahezeSSLBot 038eb5baa - Bot: Update SSL cert for pokemonwiki.info [08:28:37] PROBLEM - hoodwiki.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hoodwiki.xyz' expires in 15 day(s) (Tue 24 May 2022 08:24:38 GMT +0000). [08:29:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8eb5baa62dc5...0c68dcfc8b64 [08:29:53] [url] Comparing 8eb5baa62dc5...0c68dcfc8b64 · miraheze/ssl · GitHub | github.com [08:29:54] [02miraheze/ssl] 07MirahezeSSLBot 030c68dcf - Bot: Update SSL cert for hoodwiki.xyz [08:36:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 10.27, 9.39 [08:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.30, 9.49, 9.22 [08:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:42:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.59, 3.23 [08:44:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.37, 3.19 [08:49:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:58:14] RECOVERY - hoodwiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'hoodwiki.xyz' will expire on Sat 06 Aug 2022 07:29:47 GMT +0000. [09:24:02] RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Sat 06 Aug 2022 07:26:57 GMT +0000. [09:28:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.18, 2.77 [09:32:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.21, 2.89 [09:37:15] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.75, 1.89, 1.39 [09:37:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 10.31, 9.45 [09:39:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.55, 9.70, 9.33 [09:39:11] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.16, 1.55, 1.31 [09:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:42:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.06, 1.20, 1.95 [09:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:56:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.18, 2.78 [09:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:56:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.68, 3.19 [09:57:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.58, 10.51, 9.96 [09:58:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.33, 2.82, 2.69 [09:58:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.47, 1.67 [10:00:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.31, 3.17 [10:01:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.49, 9.32, 9.63 [10:06:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.77, 3.37 [10:06:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 10.21, 9.85 [10:08:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.68, 3.39 [10:08:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.71, 9.09, 9.50 [10:10:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 3.33, 3.30 [10:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:17:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.60, 3.46 [10:20:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 2.05, 1.76 [10:21:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:21:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 4.09, 3.67 [10:22:13] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.42, 1.83, 1.72 [10:23:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.85, 3.63 [10:24:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.69, 1.68 [10:29:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.94, 2.95, 3.32 [10:32:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.05, 10.16, 8.46 [10:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.56, 9.27, 8.35 [10:36:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.75, 1.69 [10:37:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.88, 10.30, 9.46 [10:39:23] PROBLEM - wiki.ssangyongsports.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssangyongsports.eu.org could not be found [10:39:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.11, 10.15, 9.50 [10:41:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.56, 3.41 [10:43:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 3.31, 3.33 [10:48:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 2.11, 1.86 [10:56:13] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.96, 1.95 [10:56:36] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:59:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 3.50, 2.99 [11:01:36] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:02:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.87, 4.52, 3.69 [11:03:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.27, 3.05 [11:06:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.89, 3.60 [11:09:16] RECOVERY - wiki.ssangyongsports.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.ssangyongsports.eu.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:11:36] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:14:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.09, 1.35, 1.63 [11:16:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.84, 4.19, 3.80 [11:17:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.29, 4.54, 3.69 [11:18:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.58, 3.62 [11:19:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.80, 3.51 [11:19:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 10.43, 9.63 [11:20:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.43, 9.84, 8.87 [11:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.87, 1.75 [11:21:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.41, 11.56, 10.14 [11:21:36] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.76, 1.72 [11:22:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.77, 2.75, 3.28 [11:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:23:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 10.76, 10.02 [11:23:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.91, 9.83, 9.14 [11:24:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.59, 1.66 [11:25:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.42, 2.74, 3.19 [11:26:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.65, 1.35 [11:28:21] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.06, 1.47, 1.33 [11:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.89, 11.27, 10.57 [11:40:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 11.88, 11.21 [11:44:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.47, 3.19 [11:46:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.09, 3.08 [11:50:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.84, 2.12, 1.69 [11:50:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.78, 11.55, 10.97 [11:52:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.59, 11.39, 10.05 [11:54:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.29, 1.77, 1.66 [11:54:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 10.71, 9.42 [11:54:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.86, 4.41, 2.56 [11:54:29] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.015 second response time [11:54:38] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:40] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.31, 10.28, 9.24 [11:54:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.60, 1.61 [11:56:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.40, 10.67, 9.51 [11:57:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 10.28, 8.94 [11:57:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.44, 10.45, 9.19 [11:59:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.16, 9.72, 8.89 [12:01:58] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.09, 1.73, 1.94 [12:02:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 11.44, 11.02 [12:02:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.02, 11.89, 10.37 [12:02:35] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:49] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:58] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:58] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:04] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:17] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:40] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:26] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:02] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 7.879 second response time [12:05:10] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 7.874 second response time [12:05:47] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.506 second response time [12:06:07] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.47, 1.23, 1.70 [12:06:28] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.150 second response time [12:08:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.99, 12.06, 11.39 [12:09:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.72, 10.02, 9.68 [12:10:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.29, 10.53, 9.71 [12:11:49] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.318 second response time [12:12:15] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:27] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [12:12:27] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:12:29] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.358 second response time [12:12:55] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:59] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 2.436 second response time [12:13:09] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.457 second response time [12:13:22] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.102 second response time [12:13:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 10.71, 10.10 [12:14:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.84, 9.99, 9.68 [12:14:19] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 4.136 second response time [12:14:53] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 1.402 second response time [12:15:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.78, 9.67, 9.79 [12:16:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.77, 11.82, 11.48 [12:18:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.19, 11.95, 11.99 [12:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.02, 1.73 [12:21:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.17, 11.41, 11.62 [12:23:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:24:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.97, 1.79 [12:24:27] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.084 second response time [12:24:28] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22292 bytes in 0.423 second response time [12:26:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.13, 8.17, 9.84 [12:26:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.26, 1.83, 1.30 [12:28:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.15, 1.53, 1.25 [12:28:58] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.77, 1.63, 1.46 [12:29:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.53, 2.88 [12:29:41] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.346 second response time [12:29:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.25, 8.74, 10.17 [12:29:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.37, 2.73 [12:29:52] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.678 second response time [12:30:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.70, 3.04 [12:30:29] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 1.198 second response time [12:30:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.91, 1.39, 1.39 [12:31:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.69, 3.01 [12:31:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.25, 2.76 [12:32:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:33:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.24, 2.93 [12:33:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.83, 10.01, 10.33 [12:34:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 3.02, 2.93 [12:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.51, 9.36, 10.10 [12:37:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.65, 9.29, 10.03 [12:42:22] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:46:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.85, 1.83 [12:50:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.60, 11.07, 10.50 [12:54:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.19, 1.59, 1.27 [12:56:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.81, 1.91 [12:56:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.64, 1.33 [12:57:47] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.24, 1.79, 1.39 [12:58:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.99, 11.65, 10.90 [12:58:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.69, 1.64, 1.36 [12:59:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.88, 3.37 [12:59:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.44, 1.64, 1.38 [13:00:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.18, 10.75, 10.67 [13:01:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 9.28, 5.13, 3.84 [13:02:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.35, 1.66 [13:02:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.67, 11.26, 11.96 [13:04:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.25, 9.23, 10.04 [13:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 11.71, 12.04 [13:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 11.02, 11.77 [13:10:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.97, 11.60, 11.80 [13:14:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.35, 11.58, 11.77 [13:18:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.58, 11.81, 11.79 [13:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.93, 1.96, 1.64 [13:21:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.13, 2.83 [13:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.80, 1.62 [13:22:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 11.44, 11.68 [13:23:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.26, 2.90 [13:25:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.65, 3.99 [13:25:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.23, 2.93 [13:26:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.59, 1.59 [13:27:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.75, 3.97 [13:27:35] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.69, 1.37 [13:27:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.48, 3.05 [13:27:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.87, 10.08, 9.76 [13:29:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.42, 3.83 [13:29:35] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.14, 1.40, 1.30 [13:29:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.17, 9.97, 9.75 [13:29:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.02, 2.93 [13:32:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:33:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 3.90, 3.89 [13:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 11.41, 11.36 [13:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 10.92, 11.20 [13:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:41:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.12, 9.83, 9.50 [13:42:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.53, 10.95, 10.06 [13:42:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.97, 9.70, 9.08 [13:43:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.20, 9.37, 9.36 [13:44:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.50, 9.77, 9.74 [13:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:45:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.74, 3.89 [13:46:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.76, 10.03, 9.37 [13:47:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.93, 4.13, 4.04 [13:49:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.27, 10.11, 9.54 [13:50:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.99, 1.71 [13:50:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.88, 3.48 [13:52:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.30, 1.80, 1.68 [13:52:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.58, 3.41 [13:53:05] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:53:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.21, 9.95, 9.68 [13:54:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.67, 1.64 [13:55:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.57, 3.93 [13:56:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.98, 3.63 [13:57:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.93, 4.02 [13:58:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 4.30, 3.58 [13:59:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.57, 3.88 [14:00:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.75, 3.62 [14:00:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.70, 3.44 [14:02:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.29, 3.31 [14:04:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.44, 11.09, 10.16 [14:04:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 2.79, 3.24 [14:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.26, 11.46, 10.87 [14:05:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.32, 4.33, 4.03 [14:06:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.55, 11.24, 10.33 [14:06:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.71, 1.64 [14:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.70, 11.54, 10.98 [14:08:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.50, 3.42 [14:09:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.69, 10.38, 9.94 [14:10:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.90, 11.41, 10.37 [14:12:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.08, 9.94, 10.08 [14:12:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 3.26, 3.37 [14:12:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 12.03, 11.32 [14:12:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.60, 11.43, 10.50 [14:14:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.84, 11.12, 11.09 [14:16:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.25, 10.61, 10.32 [14:16:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 4.28, 3.75 [14:16:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.55, 11.65, 10.76 [14:18:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.51, 1.96, 1.79 [14:18:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.26, 10.78, 10.54 [14:18:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.43, 11.75, 11.29 [14:19:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.87, 10.55, 9.93 [14:20:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.53, 9.75, 10.06 [14:20:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.58, 3.58 [14:20:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 11.75, 11.37 [14:21:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.46, 10.17, 9.87 [14:22:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.59, 11.46, 10.84 [14:22:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.40, 12.07, 11.54 [14:24:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.78, 10.96, 10.72 [14:24:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.78, 11.76, 11.50 [14:26:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.95, 10.43, 10.24 [14:26:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.10, 3.35 [14:26:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 12.06, 11.65 [14:28:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.80, 10.17, 10.19 [14:30:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 2.00, 1.98 [14:30:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 11.67, 11.62 [14:31:14] !log reception@mwtask111:~$ sudo -u www-data php /home/universalomega/assignImportedEdits.php --wiki=obeymewiki --import-prefix="wikia:obey-me" --from=Gemmaniac --to=Gemmaniac [14:31:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.48, 11.47, 10.71 [14:32:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 11.94, 11.73 [14:34:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.38, 10.54, 10.48 [14:34:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.60, 9.40, 10.08 [14:34:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.95, 1.60, 1.35 [14:35:53] Sounds good [14:36:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.61, 9.31, 10.02 [14:36:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.29, 1.51, 1.35 [14:38:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.80, 3.52 [14:40:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.75, 3.53 [14:41:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.30, 10.35, 10.06 [14:43:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.72, 11.25, 10.57 [14:43:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.69, 9.48, 9.76 [14:44:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.44, 1.68 [14:44:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 11.45, 10.72 [14:46:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.64, 3.54 [14:46:49] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [14:46:49] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:46:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.40, 11.04, 10.67 [14:47:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.21, 10.51, 10.11 [14:48:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 10.30, 10.18 [14:48:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.71, 3.56 [14:48:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.80, 11.72, 10.96 [14:50:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.03, 9.83, 10.02 [14:50:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.91, 1.79 [14:50:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 3.98, 3.67 [14:50:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.12, 10.84, 10.73 [14:51:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.35, 11.82, 10.72 [14:52:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.86, 1.79 [14:52:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.57, 3.56 [14:53:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.57, 11.47, 10.74 [14:54:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.17, 1.60, 1.70 [14:54:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.77, 3.62 [14:54:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.58, 9.46, 10.19 [14:56:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.38, 3.50 [14:58:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.54, 10.60, 10.30 [15:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.39, 11.35, 10.64 [15:04:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.77, 10.79, 10.53 [15:04:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.40, 3.42 [15:06:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.62, 9.60, 10.11 [15:06:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.25, 3.36, 3.40 [15:06:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.18, 1.65, 1.35 [15:08:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.03, 1.35, 1.28 [15:16:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.39, 10.73, 10.02 [15:16:44] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.86, 1.60 [15:20:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.33, 10.09, 9.95 [15:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.75, 1.60 [15:23:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.82, 11.68, 11.03 [15:24:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.62, 1.57 [15:25:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.89, 11.49, 11.03 [15:26:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 11.56, 12.00 [15:28:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.92, 11.85, 12.06 [15:30:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.70, 11.18, 11.80 [15:32:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.07, 10.69, 9.83 [15:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.50, 11.78 [15:36:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.02, 10.47, 9.99 [15:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 11.73, 11.84 [15:37:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.78, 3.50 [15:38:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.02, 10.61, 10.07 [15:40:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.29, 11.26, 10.43 [15:40:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.47, 11.67 [15:41:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.78, 9.21, 10.15 [15:42:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.34, 10.79, 10.35 [15:42:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.19, 11.53, 10.56 [15:43:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 3.28, 3.40 [15:44:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.36, 11.56, 10.69 [15:44:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.86, 11.20, 11.58 [15:48:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.07, 11.59, 11.58 [15:49:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.85, 2.17, 1.79 [15:50:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.68, 3.52 [15:50:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 11.71, 11.64 [15:51:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.19, 10.27, 10.17 [15:52:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.70, 3.56 [15:55:01] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 2.00, 1.87 [15:56:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.82, 3.60 [15:56:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.86, 11.74, 11.62 [15:57:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.40, 10.86, 10.39 [15:58:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.51, 3.52 [15:59:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.12, 9.57, 9.98 [16:00:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.49, 3.14, 3.37 [16:00:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 11.46, 11.55 [16:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.28, 11.42, 10.84 [16:02:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 11.09, 10.67 [16:04:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.28, 11.17, 10.81 [16:04:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.34, 10.18, 10.41 [16:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 11.38, 11.44 [16:08:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.67, 9.43, 10.05 [16:08:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 11.55, 11.52 [16:10:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.44, 3.31 [16:12:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.21, 3.24 [16:12:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.26, 10.45, 10.36 [16:16:52] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.89, 2.08, 1.93 [16:18:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.88, 11.69, 11.19 [16:20:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.71, 10.88, 10.95 [16:20:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.36, 3.91 [16:20:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.62, 11.48, 10.47 [16:20:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.50, 11.83, 11.52 [16:21:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.45, 3.32 [16:22:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.66, 3.95 [16:22:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.72, 11.39, 10.56 [16:24:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.73, 3.94 [16:24:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.69, 11.80, 10.81 [16:24:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.32, 1.82, 1.92 [16:26:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.01, 4.15, 4.06 [16:26:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.18, 11.45, 10.82 [16:26:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 11.73, 11.64 [16:27:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.37, 3.10 [16:28:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.25, 9.33, 10.10 [16:30:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.27, 9.58, 10.05 [16:30:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 12.04, 11.79 [16:30:47] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.42, 1.70 [16:31:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.62, 3.26 [16:32:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 10.91, 11.42 [16:33:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 3.26, 3.17 [16:33:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 2.97, 3.25 [16:34:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.63, 10.61, 10.45 [16:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.43, 11.84, 11.69 [16:36:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.18, 9.62, 10.10 [16:36:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.53, 3.93 [16:38:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.98, 5.05, 4.42 [16:40:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 11.72, 11.79 [16:42:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.40, 12.70, 12.15 [16:46:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.43, 9.57, 10.12 [16:46:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.00, 11.94, 11.99 [16:48:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 12.13, 12.06 [16:50:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 1.85, 1.58 [16:50:48] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 9.77, 3.54, 1.95 [16:52:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 10.21, 10.25 [16:52:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.81, 1.60 [16:53:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.28, 9.77, 10.08 [16:54:36] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.37, 1.65, 1.56 [16:54:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 11.63, 11.93 [16:57:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.41, 3.01 [16:58:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.99, 1.90, 1.78 [16:58:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.81, 11.90 [16:58:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.86, 3.62, 3.37 [16:59:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.08, 2.93 [16:59:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.37, 10.98, 10.12 [17:00:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.82, 1.54, 1.66 [17:00:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 11.31, 11.70 [17:00:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.28, 3.28 [17:01:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.82, 11.37, 10.37 [17:03:55] [02puppet] 07Universal-Omega synchronize pull request 03#2446: T8848: add MediaWiki Icinga docs entries - 13https://github.com/miraheze/puppet/pull/2446 [17:03:56] [url] Page not found · GitHub · GitHub | github.com [17:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 11.88, 11.81 [17:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.78, 11.67, 11.76 [17:08:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.77, 10.03, 10.16 [17:10:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 11.71, 11.73 [17:11:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.45, 10.03 [17:13:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.73, 9.60, 9.78 [17:13:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.67, 10.53, 9.87 [17:14:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.62, 11.92, 11.84 [17:15:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.50, 3.13 [17:15:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.65, 3.36 [17:16:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 11.99, 11.87 [17:17:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.08, 3.02 [17:17:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.60, 11.00, 10.43 [17:19:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.37, 3.40, 3.32 [17:20:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.94, 1.66 [17:20:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 11.65, 11.73 [17:21:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.85, 1.33 [17:21:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.40, 11.03, 10.20 [17:21:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.19, 10.53, 10.45 [17:22:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.83, 1.65 [17:22:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.56, 12.29, 11.95 [17:23:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.53, 1.27 [17:24:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.43, 3.90 [17:24:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.68, 1.62 [17:24:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 11.93, 11.88 [17:25:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.35, 10.93, 10.42 [17:27:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.96, 9.24, 9.93 [17:28:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.86, 3.94 [17:28:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.37, 12.20, 11.96 [17:31:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.92, 10.31, 10.18 [17:32:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 11.57, 11.85 [17:36:18] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.75, 1.67 [17:36:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.96, 3.48 [17:36:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.86, 11.69, 11.78 [17:37:06] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.35, 9.16, 9.91 [17:38:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.28, 11.03, 10.57 [17:38:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.63, 3.41 [17:40:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.87, 1.73 [17:40:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.47, 10.90, 10.58 [17:40:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.40, 11.43, 11.70 [17:42:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.85, 1.74 [17:44:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.31, 3.38 [17:44:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.16, 11.08, 11.43 [17:46:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.50, 3.20, 3.85 [17:46:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 11.40, 11.52 [17:48:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.82, 4.02 [17:48:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.95, 2.21, 1.90 [17:48:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.72, 3.53 [17:50:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.63, 3.93 [17:50:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.25, 9.81, 10.14 [17:52:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.54, 9.81, 10.16 [17:52:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.54, 3.49 [17:52:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 11.64, 11.54 [17:53:05] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:54:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.34, 4.60, 4.26 [17:54:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.51, 11.50 [17:56:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.95, 1.94 [17:56:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 11.76, 11.58 [18:00:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.70, 11.30, 11.50 [18:04:00] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=criticalrolewiki --dfn-only (START) [18:04:18] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=criticalrolewiki --dfn-only (END - exit=0) [18:04:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.55, 11.51 [18:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.43, 10.86, 10.22 [18:05:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.35, 9.87, 8.85 [18:06:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.71, 2.06, 1.95 [18:06:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.45, 10.31, 9.95 [18:06:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.10, 11.07, 10.41 [18:07:18] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.66, 10.75, 9.29 [18:08:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:10] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 10.96, 10.23 [18:08:26] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:08:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 2.91, 3.34 [18:08:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.38, 11.12, 10.50 [18:08:56] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:29] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 11.32, 9.89 [18:11:31] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=criticalrolewiki --namespace=14 (START) [18:11:36] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:43] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.15, 10.65, 9.53 [18:12:08] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:12:27] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:28] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:35] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:56] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.41, 11.41, 9.95 [18:14:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.56, 11.72, 10.96 [18:15:25] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=criticalrolewiki --namespace=14 (END - exit=0) [18:16:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:29] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 9.033 second response time [18:16:37] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.912 second response time [18:16:56] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:10] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:21] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.63, 12.07, 10.67 [18:17:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.74, 11.65, 10.39 [18:18:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.25, 11.92, 11.24 [18:18:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.60, 11.70, 11.20 [18:19:04] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 9.314 second response time [18:19:06] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:14] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.726 second response time [18:19:24] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 7.510 second response time [18:19:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.56, 11.84, 10.76 [18:19:44] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:48] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.476 second response time [18:20:02] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.399 second response time [18:20:33] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:49] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:52] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:53] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:00] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 3.45, 3.08 [18:21:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 4.07, 3.58 [18:22:42] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.689 second response time [18:22:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 11.86, 12.00 [18:22:54] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.625 second response time [18:23:05] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.253 second response time [18:23:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.51, 11.94, 11.04 [18:24:00] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:04] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [18:24:25] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.94, 12.27, 12.13 [18:25:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.58, 3.25 [18:25:16] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw121 mw122 [18:25:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.66, 10.49, 10.62 [18:25:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.02, 8.81, 9.72 [18:25:36] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:39] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:50] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:58] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.848 second response time [18:26:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.84, 11.95, 11.45 [18:26:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.99, 12.06, 11.53 [18:27:01] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.540 second response time [18:27:01] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 0.662 second response time [18:27:09] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:11] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:19] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:32] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.020 second response time [18:27:52] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.367 second response time [18:28:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 11.88, 11.50 [18:28:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.71, 11.82, 11.52 [18:29:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.44, 11.25, 11.58 [18:29:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.01, 4.10, 3.51 [18:29:11] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [18:30:17] Soooo.... CLG wiki has an outage... again. [18:31:31] Also, I've officially discovered just now what it was like to chat online back in the old days... Man, I was born way too late for this stuff. [18:31:56] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.685 second response time [18:32:11] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.285 second response time [18:32:26] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.293 second response time [18:32:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.63, 11.49, 11.95 [18:33:24] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 6.175 second response time [18:33:32] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.424 second response time [18:33:36] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.759 second response time [18:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.15, 11.88, 12.02 [18:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 11.03, 11.69 [18:37:52] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.199 second response time [18:37:55] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.659 second response time [18:37:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.028 second response time [18:37:57] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 4.385 second response time [18:38:12] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.648 second response time [18:38:14] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.787 second response time [18:38:15] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.792 second response time [18:38:17] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.167 second response time [18:38:20] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.472 second response time [18:39:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 9.52, 10.16 [18:40:07] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:40:50] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 0.873 second response time [18:42:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.46, 8.41, 9.72 [18:44:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.89, 8.07, 9.78 [18:48:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.40, 11.21, 11.21 [18:50:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.42, 9.35, 10.18 [18:50:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3b874764bf65 [18:50:41] [02miraheze/mw-config] 07Universal-Omega 033b87476 - Guarantee that server is always a string [18:50:43] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [18:50:43] [url] Page not found · GitHub · GitHub | github.com [18:51:36] [02mw-config] 07Universal-Omega opened pull request 03#4672: Guarantee that server is always a string - 13https://github.com/miraheze/mw-config/pull/4672 [18:51:37] [url] Page not found · GitHub · GitHub | github.com [18:52:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.10, 10.53, 10.06 [18:52:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 11.78, 11.45 [18:52:43] miraheze/mw-config - Universal-Omega the build passed. [18:53:38] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.34, 11.20, 10.36 [18:54:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 12.03, 11.58 [18:56:31] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.78, 10.85, 10.33 [18:56:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.80, 11.97, 11.62 [18:58:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.02, 10.09, 10.10 [18:59:30] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:31] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:50] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:53] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:35] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 9.190 second response time [19:01:35] [02mw-config] 07Universal-Omega synchronize pull request 03#4672: Guarantee that server is always a string - 13https://github.com/miraheze/mw-config/pull/4672 [19:01:36] [url] Page not found · GitHub · GitHub | github.com [19:01:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b874764bf65...8f36c66d7348 [19:01:38] [url] Comparing 3b874764bf65...8f36c66d7348 · miraheze/mw-config · GitHub | github.com [19:01:38] [02miraheze/mw-config] 07Universal-Omega 038f36c66 - Update MirahezeFunctions.php [19:01:54] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.403 second response time [19:01:58] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 6.276 second response time [19:02:10] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.262 second response time [19:02:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 11.73, 11.53 [19:02:46] [02mw-config] 07Universal-Omega edited pull request 03#4672: Guarantee that server is always a string - 13https://github.com/miraheze/mw-config/pull/4672 [19:02:47] [url] Page not found · GitHub · GitHub | github.com [19:02:47] miraheze/mw-config - Universal-Omega the build passed. [19:03:33] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.486 second response time [19:05:01] [02mw-config] 07Universal-Omega closed pull request 03#4672: Guarantee that server is always a string - 13https://github.com/miraheze/mw-config/pull/4672 [19:05:01] [url] Page not found · GitHub · GitHub | github.com [19:05:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/52be0a6993c9...f98faa9998e8 [19:05:03] [url] Comparing 52be0a6993c9...f98faa9998e8 · miraheze/mw-config · GitHub | github.com [19:05:04] [02miraheze/mw-config] 07Universal-Omega 03f98faa9 - Guarantee that server is always a string (#4672) [19:05:05] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:05:07] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:05:07] ... [19:05:41] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [19:05:43] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [19:06:04] miraheze/mw-config - Universal-Omega the build passed. [19:06:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:06] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [19:07:26] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 19s [19:07:33] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 14 minutes ago with 0 failures [19:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 11.15, 11.42 [19:09:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.85, 2.16, 1.50 [19:09:33] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:10:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 11.87, 11.66 [19:11:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.02, 1.72, 1.42 [19:13:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.38, 1.32 [19:16:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.63, 10.05, 9.73 [19:17:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.82, 2.60, 1.85 [19:18:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 11.39, 11.64 [19:19:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.01, 1.95, 1.70 [19:20:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.07, 10.15, 9.87 [19:20:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.25, 11.67, 11.71 [19:20:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.83, 3.51, 1.86 [19:21:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.17, 1.70, 1.64 [19:21:58] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:00] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:16] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 11.25, 11.55 [19:23:00] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:57] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 2.888 second response time [19:23:59] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.243 second response time [19:24:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 3.593 second response time [19:24:47] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [19:24:47] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:25:01] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 3.709 second response time [19:26:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.77, 11.67 [19:26:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.39, 1.48, 1.48 [19:28:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.54, 10.44, 9.95 [19:28:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 11.57, 11.60 [19:29:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.43, 10.29, 9.22 [19:31:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.26, 9.95, 9.23 [19:32:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.57, 10.60, 9.80 [19:32:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 11.56, 11.55 [19:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.48, 9.79, 9.93 [19:34:45] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:35:22] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:49] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.508 second response time [19:37:22] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 2.940 second response time [19:38:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.71, 9.57, 9.64 [19:38:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.34, 11.77, 11.79 [19:42:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 10.33, 9.90 [19:44:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.88, 10.05, 9.84 [19:46:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.66, 10.07, 9.84 [19:48:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.59, 11.14, 10.25 [19:50:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.52, 11.16, 10.39 [19:50:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 10.44, 10.08 [19:51:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.59, 10.13, 9.63 [19:52:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.37, 9.85, 10.01 [19:53:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.14, 9.93, 9.63 [19:54:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.20, 11.72, 11.51 [19:56:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.81, 2.33, 1.56 [19:56:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 11.17, 11.34 [19:58:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 10.17, 12.21, 5.91 [20:00:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.20, 9.30, 9.88 [20:00:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.31, 1.79, 1.51 [20:02:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.43, 1.63, 1.48 [20:02:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.25, 11.57, 11.39 [20:07:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:08] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:05] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.517 second response time [20:09:06] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.314 second response time [20:11:58] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.98, 10.85, 10.35 [20:12:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.41, 11.85, 11.77 [20:12:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.96, 10.98, 9.86 [20:13:59] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 1.455 second response time [20:16:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.72, 9.99, 9.74 [20:18:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.32, 9.68, 10.07 [20:22:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.52, 0.83, 1.87 [20:24:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 11.60, 11.43 [20:26:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.86, 0.92, 1.66 [20:26:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 11.28, 11.33 [20:27:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.40, 1.23 [20:28:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 11.63, 11.43 [20:29:11] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.84, 1.17, 1.17 [20:32:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.58, 10.22, 9.78 [20:37:19] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:38:42] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.41, 9.81, 9.78 [20:39:25] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.015 second response time [20:44:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.29, 10.42, 10.01 [20:46:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.54, 9.91, 9.87 [20:46:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.39, 10.25, 9.80 [20:46:55] [02puppet] 07Universal-Omega opened pull request 03#2530: firejail: blacklist /var/www/.ssh - 13https://github.com/miraheze/puppet/pull/2530 [20:46:56] [url] Page not found · GitHub · GitHub | github.com [20:50:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.70, 9.98, 9.82 [20:52:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.91, 10.29, 9.06 [20:54:01] [02puppet] 07Universal-Omega opened pull request 03#2531: ImageMagick policy: add map resource limit - 13https://github.com/miraheze/puppet/pull/2531 [20:54:02] [url] Page not found · GitHub · GitHub | github.com [20:54:32] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:56:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.73, 9.38, 9.02 [20:57:15] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:12] [02puppet] 07paladox closed pull request 03#2530: firejail: blacklist /var/www/.ssh - 13https://github.com/miraheze/puppet/pull/2530 [20:58:13] [url] Page not found · GitHub · GitHub | github.com [20:58:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d5a85b6adafa...560e664b008f [20:58:14] [url] Comparing d5a85b6adafa...560e664b008f · miraheze/puppet · GitHub | github.com [20:58:15] [02miraheze/puppet] 07Universal-Omega 03560e664 - firejail: blacklist /var/www/.ssh (#2530) [20:58:23] [02puppet] 07paladox closed pull request 03#2531: ImageMagick policy: add map resource limit - 13https://github.com/miraheze/puppet/pull/2531 [20:58:24] ... [20:58:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/560e664b008f...85867491509d [20:58:26] [url] Comparing 560e664b008f...85867491509d · miraheze/puppet · GitHub | github.com [20:58:26] [02miraheze/puppet] 07Universal-Omega 038586749 - ImageMagick policy: add map resource limit (#2531) [20:59:14] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 5.204 second response time [21:06:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.64, 10.21, 9.38 [21:06:21] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=criticalrolewiki (START) [21:06:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:08:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.70, 10.27, 9.83 [21:10:47] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.42, 9.98, 9.78 [21:11:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.54, 9.99, 9.59 [21:17:06] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 10.42, 9.87 [21:17:17] !log reboot mw111 [21:17:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.94, 2.97, 1.03 [21:19:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.94, 10.45, 9.96 [21:20:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.79, 9.61, 9.72 [21:22:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.56, 3.97 [21:22:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 8.10, 3.60 [21:23:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.19, 11.24, 10.22 [21:24:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:25:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 10.50, 10.08 [21:26:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.29, 10.02, 5.49 [21:27:16] [02puppet] 07paladox closed pull request 03#2514: DO NOT MERGE - 13https://github.com/miraheze/puppet/pull/2514 [21:27:16] [url] Page not found · GitHub · GitHub | github.com [21:27:18] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [21:27:18] [url] Page not found · GitHub · GitHub | github.com [21:27:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [21:28:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:28:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.37, 9.97, 6.04 [21:30:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.24, 3.58 [21:31:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.64, 10.16, 10.12 [21:32:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.90, 10.52, 10.07 [21:32:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.97, 11.11, 7.41 [21:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.52, 10.12, 10.00 [21:34:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.91, 3.84 [21:35:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:36:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.43, 4.60, 4.11 [21:42:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.37, 11.96, 9.52 [21:44:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 10.73, 9.49 [21:44:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.87, 3.96 [21:44:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 10.97, 9.46 [21:46:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 10.19, 9.45 [21:46:11] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=criticalrolewiki (END - exit=2) [21:46:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.84, 4.64, 4.23 [21:46:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:55] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/recountCategories.php --wiki=criticalrolewiki --mode=pages (END - exit=0) [21:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:48] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/cleanupEmptyCategories.php --wiki=criticalrolewiki --mode=remove (END - exit=0) [21:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 11.36, 9.90 [21:50:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.35, 10.08 [21:53:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.85, 10.68, 10.19 [21:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.12, 1.50, 1.96 [21:56:16] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [21:56:16] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:56:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.16, 9.80, 9.85 [21:57:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.22, 10.08, 10.10 [22:02:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.03, 11.10, 10.08 [22:02:13] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.26, 1.70 [22:04:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.17, 9.57, 9.65 [22:06:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.01, 3.04, 3.97 [22:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.81, 10.46, 10.00 [22:08:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.45, 4.00 [22:08:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.96, 9.55, 9.71 [22:10:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.66, 2.80, 3.69 [22:11:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.97, 3.04, 3.95 [22:13:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.93, 3.62, 4.03 [22:14:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.38, 3.69 [22:15:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:15:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.33, 3.87 [22:18:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.40, 3.64 [22:18:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.27, 3.22, 4.00 [22:20:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.76, 2.11, 1.80 [22:22:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.66, 1.95, 1.78 [22:22:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.42, 3.56 [22:22:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.44, 3.90 [22:25:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.05, 4.01, 3.87 [22:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:28:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.34, 1.63, 1.69 [22:29:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.97, 3.89 [22:31:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.50, 4.35, 4.02 [22:32:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 9.90, 9.37 [22:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.97, 9.71, 9.35 [22:37:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.58, 3.66, 3.92 [22:38:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.00, 3.73, 3.91 [22:40:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.53, 9.94, 9.48 [22:41:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:42:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.07, 9.65 [22:42:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.89, 9.83, 9.50 [22:43:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 2.56, 3.35 [22:43:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.89, 9.36, 9.45 [22:46:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.74, 1.64 [22:48:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.14, 2.74, 3.30 [22:48:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.77, 2.09, 1.78 [22:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:58:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.98, 1.91 [22:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:59:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.33, 3.29 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:13] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.99, 2.02, 1.95 [23:03:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.08, 3.15, 3.23 [23:04:12] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 2.00, 1.96 [23:04:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.93, 10.39, 9.86 [23:06:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 2.17, 2.03 [23:06:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 10.07, 9.82 [23:07:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.42, 3.32 [23:09:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.14, 3.23, 3.26 [23:11:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.60, 9.79, 8.78 [23:12:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.66, 9.55, 9.02 [23:13:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 9.27, 6.13, 4.39 [23:13:12] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:12] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:17] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.90, 9.02, 8.62 [23:13:56] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:58] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.73, 6.65, 4.58 [23:14:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 8.73, 8.79 [23:14:42] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:43] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:58] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:03] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 9.77, 8.51 [23:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:16:44] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 2.532 second response time [23:16:57] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 2.330 second response time [23:18:00] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.360 second response time [23:18:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.308 second response time [23:18:35] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:41] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:45] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:45] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.360 second response time [23:19:21] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.563 second response time [23:19:21] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.504 second response time [23:19:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.91, 9.10, 8.55 [23:19:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 10.83, 10.18 [23:19:36] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.757 second response time [23:19:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.23, 10.30, 8.91 [23:20:31] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.468 second response time [23:20:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 6.416 second response time [23:20:44] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.220 second response time [23:20:47] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.794 second response time [23:20:50] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 3.848 second response time [23:20:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.91, 1.58, 1.14 [23:21:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.48, 9.25, 8.70 [23:22:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.43, 1.44, 1.14 [23:23:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 10.91, 10.36 [23:25:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.30, 10.55 [23:25:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 11.07, 9.57 [23:26:00] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [23:27:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 11.11, 10.56 [23:27:06] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1623s [23:27:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.23, 11.03, 9.74 [23:27:47] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File [23:27:47] ib/python3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query vedopedia.witches-empire.com. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:27:55] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1673s [23:28:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.23, 9.23, 9.33 [23:32:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.83, 10.51, 9.69 [23:32:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.04, 8.87, 9.85 [23:34:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.81, 9.81, 9.53 [23:42:11] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.17, 1.37, 1.94 [23:45:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:48:11] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 1.81, 1.91 [23:50:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:51:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 10.72, 10.09 [23:52:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.81, 10.14, 9.94 [23:54:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.82, 1.93 [23:57:46] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp20.miraheze.org - CNAME OK