[00:00:20] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.49, 9.50, 9.34 [00:02:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.76, 9.08, 9.26 [00:02:54] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us [00:02:54] thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:03:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.97, 8.88, 9.13 [00:03:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.74, 9.19 [00:06:12] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:06:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 9.93, 9.42 [00:09:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.11, 9.44, 9.33 [00:12:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.47, 8.98, 9.17 [00:13:49] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:14:59] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.95, 9.82, 9.46 [00:16:10] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.03, 1.43, 1.85 [00:17:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.99, 9.86, 9.53 [00:20:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.21, 9.16, 9.69 [00:20:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.77, 9.54 [00:23:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.45, 9.85, 9.63 [00:24:57] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.17, 1.36, 1.63 [00:26:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.52, 10.89, 10.18 [00:26:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.86, 9.50, 9.24 [00:29:37] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.44, 10.12, 9.78 [00:29:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.58, 8.68, 8.99 [00:32:23] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:18] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [00:35:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.85, 9.39, 9.12 [00:41:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.14, 9.29, 9.75 [00:41:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.30, 9.89, 9.61 [00:46:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 9.50, 9.93 [00:47:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.18, 9.97, 9.81 [00:50:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.69, 9.67, 9.74 [00:53:11] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 15.60, 12.51, 10.84 [00:53:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.06, 6.69, 8.09 [00:53:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [00:56:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:59:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.53, 9.00, 9.85 [01:02:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.88, 10.45, 10.26 [01:03:38] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u [01:03:38] ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:07:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.43, 9.54, 9.93 [01:10:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.15, 10.24, 10.08 [01:13:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 9.29, 9.72 [01:19:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.79, 10.12, 9.46 [01:20:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.89, 10.20, 9.04 [01:22:37] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:54] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.06, 7.96, 9.89 [01:23:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [01:24:36] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:16] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:46] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.64, 9.17, 9.32 [01:26:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.85, 10.62, 10.02 [01:27:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.02, 8.56, 8.84 [01:29:18] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:22] PROBLEM - cp31 Stunnel Http for mw112 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:38] PROBLEM - cp30 Stunnel Http for mw112 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:42] PROBLEM - cp20 Stunnel Http for mw112 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:29:51] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.240 second response time [01:30:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:31:16] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.147 second response time [01:31:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.570 second response time [01:32:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.38, 7.11, 8.27 [01:32:20] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.540 second response time [01:32:33] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22194 bytes in 5.398 second response time [01:32:35] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.591 second response time [01:32:38] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [01:32:50] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.577 second response time [01:32:52] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.142 second response time [01:33:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.92, 8.33, 8.89 [01:37:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.56, 7.89, 8.27 [01:39:15] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:26] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:39:49] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:41:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [01:42:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.397 second response time [01:42:49] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.329 second response time [01:43:08] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.949 second response time [01:43:08] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:43:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [01:44:35] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:44:59] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:46:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.77, 8.65, 9.86 [01:47:06] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:49:31] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.281 second response time [01:50:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:50:53] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.231 second response time [01:51:15] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.65, 1.93, 1.48 [01:51:19] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.598 second response time [01:52:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.41, 8.95, 9.75 [01:53:08] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.945 second response time [01:53:08] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 1.321 second response time [01:53:23] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:55:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.11, 8.95, 9.57 [01:55:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:58:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 8.56, 9.31 [01:58:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.00, 8.55, 8.43 [02:03:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.10, 9.81, 9.57 [02:04:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 8.99, 8.61 [02:04:30] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:05:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 9.02, 8.53 [02:06:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 9.75, 9.46 [02:06:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.30, 9.02, 8.67 [02:07:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.32, 9.64, 8.85 [02:09:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.19, 9.48, 8.92 [02:15:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.37, 9.34, 9.22 [02:20:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.03, 9.46, 9.72 [02:21:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:21:34] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407201290130615 seconds [02:22:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.55, 9.85, 9.85 [02:28:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.47, 9.59 [02:31:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.18, 9.79, 9.62 [02:34:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.58, 9.90, 9.71 [02:35:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.09, 8.76, 8.78 [02:36:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 9.37, 9.61 [02:38:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.21, 9.08, 8.90 [02:39:46] [02puppet] 07Universal-Omega opened pull request 03#2371: Revert "Revert "Revert "absent manageInactiveWikis cron""" - 13https://github.com/miraheze/puppet/pull/2371 [02:41:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.60, 9.37, 9.02 [02:42:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.32, 9.99, 9.77 [02:44:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.91, 9.30, 9.02 [02:47:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.44, 9.54, 9.15 [02:48:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.18, 8.98, 9.47 [02:50:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.35, 8.82, 8.95 [02:51:24] RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:54:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.36, 9.90, 9.37 [02:57:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.57, 9.62, 9.33 [03:00:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.92, 10.81, 9.85 [03:00:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:04:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:06:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.80, 9.81, 9.84 [03:09:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.16, 9.38, 9.94 [03:10:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.84, 8.90, 8.73 [03:11:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.45, 9.48, 9.25 [03:12:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.84, 9.88, 10.05 [03:13:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.33, 9.16, 8.87 [03:14:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 9.86, 9.44 [03:14:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.01, 9.49, 9.56 [03:17:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.64, 10.15, 9.63 [03:20:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.36, 9.73, 9.60 [03:20:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.25, 9.88, 9.84 [03:28:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.09, 9.67, 9.57 [03:32:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.97, 10.28, 9.95 [03:34:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.74, 9.31, 9.51 [03:39:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.59, 9.97, 9.66 [03:41:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.05, 9.95, 9.28 [03:43:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 9.66, 9.90 [03:44:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.80, 9.54, 9.23 [03:44:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:48:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.36, 10.74, 10.22 [03:50:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.02, 9.46, 9.25 [03:51:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:53:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.59, 9.49, 9.29 [03:54:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 9.85, 9.86 [04:00:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.24, 9.24, 9.74 [04:02:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.17, 9.41, 9.57 [04:05:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 9.47, 9.58 [04:05:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.92, 9.19, 9.91 [04:09:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.37, 7.91, 8.35 [04:10:14] [02mw-config] 07Universal-Omega closed pull request 03#4404: Set $wgMainStash for betaheze - 13https://github.com/miraheze/mw-config/pull/4404 [04:10:18] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [04:10:19] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [04:10:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/038b6ffd44af [04:10:23] [02miraheze/mw-config] 07Universal-Omega 03038b6ff - Set $wgMainStash for betaheze [04:10:24] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://github.com/miraheze/mw-config [04:10:26] [02mw-config] 07Universal-Omega reopened pull request 03#4404: Set $wgMainStash for betaheze - 13https://github.com/miraheze/mw-config/pull/4404 [04:10:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.24, 10.19, 9.64 [04:11:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.45, 10.05, 10.02 [04:12:56] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us [04:12:56] thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:19:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.88, 9.98, 9.56 [04:22:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.97, 9.74, 9.05 [04:25:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.93, 9.37, 9.52 [04:26:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:31:25] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.41, 10.01, 9.74 [04:31:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 9.29, 9.08 [04:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:34:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.27, 9.92, 9.77 [04:34:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.40, 9.36, 9.14 [04:39:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 9.32, 9.78 [04:40:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.39, 9.65, 9.63 [04:42:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.84, 10.35, 10.11 [04:42:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.12, 8.83, 9.31 [04:43:18] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [04:45:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.97, 9.99, 9.99 [04:48:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 9.24, 9.05 [04:51:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.82, 9.27, 9.28 [04:51:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.80, 8.76, 8.88 [04:54:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.09, 9.20 [04:56:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.52, 9.42, 9.29 [04:59:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.38, 9.22, 9.29 [05:02:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 9.08, 9.22 [05:02:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.56, 9.50, 9.43 [05:08:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.45, 10.15, 9.74 [05:10:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.20, 11.46, 10.08 [05:11:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.86, 10.96, 9.81 [05:13:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.36, 9.79, 9.55 [05:16:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 9.62, 10.04, 9.71 [05:25:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.94, 9.53, 9.86 [05:31:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.17, 10.65, 10.14 [05:31:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.80, 9.38, 9.77 [05:36:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.78, 9.43, 9.79 [05:36:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.76, 9.71, 9.99 [05:39:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.18, 9.37, 9.78 [05:42:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.47, 8.68, 9.69 [05:42:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.77, 8.83, 9.48 [05:44:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.22, 8.90, 9.73 [05:45:08] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.04, 7.37, 8.48 [05:47:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 8.84, 9.41 [05:48:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:50:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.10, 7.22, 8.36 [05:50:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.67, 8.42, 9.17 [05:53:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.30, 7.18, 8.25 [05:53:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:55:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.07, 9.77, 9.45 [05:56:30] [02mw-config] 07lens0021 synchronize pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411 [05:57:31] miraheze/mw-config - lens0021 the build passed. [05:58:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.39, 9.34 [05:58:46] [02mw-config] 07lens0021 commented on pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411#issuecomment-1017145818 [05:58:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 9.52, 9.40 [05:59:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 9.38, 8.95 [06:00:49] [02mw-config] 07Universal-Omega reviewed pull request 03#4411 commit - 13https://github.com/miraheze/mw-config/pull/4411#discussion_r788372293 [06:01:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 9.22, 9.29 [06:01:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 9.74, 9.50 [06:02:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.24, 8.03, 8.51 [06:05:25] [02mw-config] 07lens0021 opened pull request 03#4415: extension-list: Sort lines (case sensitive) - 13https://github.com/miraheze/mw-config/pull/4415 [06:06:03] [02mw-config] 07lens0021 reviewed pull request 03#4411 commit - 13https://github.com/miraheze/mw-config/pull/4411#discussion_r788374473 [06:06:25] miraheze/mw-config - lens0021 the build passed. [06:07:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 9.83, 9.66 [06:08:12] [02mw-config] 07Universal-Omega commented on pull request 03#4415: extension-list: Sort lines (case sensitive) - 13https://github.com/miraheze/mw-config/pull/4415#issuecomment-1017150177 [06:08:29] [02mw-config] 07Universal-Omega closed pull request 03#4415: extension-list: Sort lines (case sensitive) - 13https://github.com/miraheze/mw-config/pull/4415 [06:08:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/197bd8091731...ea226ca61190 [06:08:32] [02miraheze/mw-config] 07lens0021 03ea226ca - extension-list: Sort lines (case sensitive) (#4415) [06:08:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:37] [02mw-config] 07Universal-Omega reviewed pull request 03#4411 commit - 13https://github.com/miraheze/mw-config/pull/4411#discussion_r788375892 [06:09:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.11, 10.37, 9.89 [06:09:42] miraheze/mw-config - Universal-Omega the build passed. [06:10:31] [02mw-config] 07lens0021 synchronize pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411 [06:10:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.77, 9.52, 8.89 [06:11:01] [02miraheze/mw-config] 07Universal-Omega pushed 0360 commits to 03Universal-Omega-patch-5 [+0/-2/±75] 13https://github.com/miraheze/mw-config/compare/499036447d16...e55862f0f077 [06:11:02] [02miraheze/mw-config] 07Universal-Omega 03e55862f - Merge branch 'master' into Universal-Omega-patch-5 [06:11:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.96, 9.55, 9.97 [06:11:04] [02mw-config] 07Universal-Omega synchronize pull request 03#4374: Re-enable score with shellbox - 13https://github.com/miraheze/mw-config/pull/4374 [06:11:33] [02mw-config] 07lens0021 synchronize pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411 [06:11:34] miraheze/mw-config - lens0021 the build passed. [06:12:04] miraheze/mw-config - Universal-Omega the build passed. [06:12:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.69, 9.59, 9.67 [06:12:30] miraheze/mw-config - lens0021 the build passed. [06:13:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:13:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.68, 8.93, 8.75 [06:19:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.02, 7.95, 8.38 [06:20:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.91, 9.66, 9.08 [06:23:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.52, 9.30, 9.02 [06:26:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.31, 7.84, 8.47 [06:26:24] !log [@mwtask111] starting deploy of {'config': True} to scsvg [06:26:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:37] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 12s [06:26:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.79, 9.85, 9.64 [06:32:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.87, 9.97, 8.76 [06:34:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.06, 9.47, 9.00 [06:35:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.48, 8.91, 8.56 [06:35:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 9.08, 8.82 [06:35:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 8.29, 8.15 [06:35:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.48, 9.88, 9.94 [06:36:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.94, 8.98, 8.91 [06:38:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 9.88, 9.17 [06:38:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.43, 10.45, 10.14 [06:40:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.79, 8.46, 8.50 [06:41:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.77, 9.12, 8.47 [06:41:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 9.32, 9.73 [06:43:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.46, 9.02, 9.10 [06:43:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.18, 8.91, 8.23 [06:43:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 8.62, 8.38 [06:46:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.99, 8.16, 8.06 [06:48:31] [02puppet] 07Reception123 closed pull request 03#2370: Add webVideoTranscodePrioritized to high intensity jobs - 13https://github.com/miraheze/puppet/pull/2370 [06:48:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:48:33] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/9a5ce7ef7fea...e4cd9416a8c0 [06:48:34] [02miraheze/puppet] 07Universal-Omega 03e4cd941 - Add webVideoTranscodePrioritized to high intensity jobs (#2370) [06:49:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.34, 8.06, 8.16 [06:53:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.01, 9.56, 8.72 [06:53:18] [02mw-config] 07lens0021 opened pull request 03#4416: test: Check if extension-list alphabetically is ordered - 13https://github.com/miraheze/mw-config/pull/4416 [06:53:25] [02mw-config] 07lens0021 edited pull request 03#4416: test: Check if extension-list is ordered alphabetically - 13https://github.com/miraheze/mw-config/pull/4416 [06:54:18] miraheze/mw-config - lens0021 the build passed. [06:54:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.26, 7.86, 8.42 [06:55:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.29, 8.74, 8.54 [06:58:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.98, 8.45, 8.52 [06:58:16] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ea226ca61190...e686337a10cf [06:58:17] [02miraheze/mw-config] 07Reception123 03e686337 - disable createaccount on view for minecraftjapanwiki per T8644 [06:58:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:59:06] [02mw-config] 07Universal-Omega commented on pull request 03#4416: test: Check if extension-list is ordered alphabetically - 13https://github.com/miraheze/mw-config/pull/4416#issuecomment-1017174087 [06:59:14] [02mw-config] 07Universal-Omega closed pull request 03#4416: test: Check if extension-list is ordered alphabetically - 13https://github.com/miraheze/mw-config/pull/4416 [06:59:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e686337a10cf...105a2b4ad870 [06:59:17] [02miraheze/mw-config] 07lens0021 03105a2b4 - test: Check if extension-list is ordered alphabetically (#4416) [06:59:21] miraheze/mw-config - Reception123 the build passed. [07:00:20] miraheze/mw-config - Universal-Omega the build passed. [07:01:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.10, 8.37, 8.48 [07:01:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.33, 8.40, 8.50 [07:03:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:05:53] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=lgbtawiki --username-prefix wikia:lgbta --report 1 --no-updates /home/reception/lgbta_pages_full.xml (START) [07:05:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.27, 10.00, 9.50 [07:09:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 9.36, 9.35 [07:10:32] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/8892ac8c4622...59015993cdab [07:10:34] [02miraheze/dns] 07Reception123 035901599 - add dragonquestwiki.com zone [07:10:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.75, 10.05, 9.42 [07:11:07] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/59015993cdab...d2c95cc5afe4 [07:11:08] [02miraheze/dns] 07Reception123 03d2c95cc - add crocwiki.com zone [07:11:51] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/d2c95cc5afe4...fbcad943a49f [07:11:53] [02miraheze/dns] 07Reception123 03fbcad94 - add experimyco.com zone [07:13:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.03, 8.65, 8.41 [07:13:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.85, 9.46 [07:13:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.52, 9.29, 8.86 [07:14:00] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.38, 8.60, 8.39 [07:16:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.68, 10.47, 9.40 [07:18:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.84, 9.24, 8.51 [07:18:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 10.15, 9.71 [07:19:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.93, 10.23, 9.18 [07:19:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.45, 10.57, 9.88 [07:21:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.05, 9.62, 9.10 [07:22:34] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.84, 8.19, 8.40 [07:23:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.92, 9.45, 9.60 [07:24:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.67, 9.20 [07:25:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.27, 9.78, 9.73 [07:25:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.96, 9.42, 9.39 [07:26:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.29, 8.09, 8.31 [07:27:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.25, 8.68, 8.90 [07:27:55] !log [@mwtask111] starting deploy of {'config': True} to scsvg [07:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:15] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 20s [07:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:29:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.22, 9.88, 9.66 [07:32:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.07, 8.42, 9.15 [07:33:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:34:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 9.47, 9.38 [07:37:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=polandballwiki /home/reception/polandballdel.txt -r Requested at [[phab:T8645|T8645]] (START) [07:37:15] https://meta.miraheze.org/wiki/phab:T8645 [07:37:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=polandballwiki /home/reception/polandballdel.txt -r Requested at [[phab:T8645|T8645]] (END - exit=32512) [07:37:17] https://meta.miraheze.org/wiki/phab:T8645 [07:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:26] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=polandballwiki /home/reception/polandballdel.txt -r=Requested at [[phab:T8645|T8645]] (START) [07:37:26] https://meta.miraheze.org/wiki/phab:T8645 [07:37:27] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=polandballwiki /home/reception/polandballdel.txt -r=Requested at [[phab:T8645|T8645]] (END - exit=32512) [07:37:27] https://meta.miraheze.org/wiki/phab:T8645 [07:37:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 9.18, 9.27 [07:38:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.18, 7.60, 8.31 [07:40:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.16, 9.21 [07:41:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:41:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.49, 9.51, 9.33 [07:44:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 8.82, 8.39 [07:44:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.71, 9.08, 9.22 [07:46:57] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.64, 8.19, 8.21 [07:48:39] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://github.com/miraheze/ssl/compare/fa1faa05c9aa...947fc5ef6a11 [07:48:40] [02miraheze/ssl] 07Reception123 03de412d8 - add dragonquestwiki.com cert [07:48:42] [02miraheze/ssl] 07Reception123 039839f77 - add crocwiki.com cert [07:48:43] [02miraheze/ssl] 07Reception123 03947fc5e - add experimyco.com cert [07:50:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 9.77, 9.39 [07:51:27] [02ssl] 07Reception123 closed pull request 03#473: remove rangpurpedia.xyz cert (expired domain) - 13https://github.com/miraheze/ssl/pull/473 [07:51:28] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±2] 13https://github.com/miraheze/ssl/compare/947fc5ef6a11...966c32072eeb [07:51:30] [02miraheze/ssl] 07Reception123 03966c320 - remove rangpurpedia.xyz cert (expired domain) (#473) [07:51:31] [02ssl] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/ssl [07:51:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:51:33] [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-1 [07:51:34] [02ssl] 07Reception123 closed pull request 03#475: Remove sportsanalytics.wiki - 13https://github.com/miraheze/ssl/pull/475 [07:51:36] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/966c32072eeb...96da87acb387 [07:51:37] [02miraheze/ssl] 07Universal-Omega 0396da87a - Remove sportsanalytics.wiki (#475) [07:51:39] [02ssl] 07Reception123 closed pull request 03#476: Remove www.wikimicrofinanza.it - 13https://github.com/miraheze/ssl/pull/476 [07:51:40] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/96da87acb387...b00c782af2aa [07:51:42] [02miraheze/ssl] 07MacFan4000 03b00c782 - Remove www.wikimicrofinanza.it (#476) [07:51:43] [02ssl] 07Reception123 closed pull request 03#478: Remove linkwiki.cf - 13https://github.com/miraheze/ssl/pull/478 [07:51:45] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/b00c782af2aa...7e72a5740091 [07:51:46] [02miraheze/ssl] 07Universal-Omega 037e72a57 - Remove linkwiki.cf (#478) [07:51:48] [02ssl] 07Reception123 closed pull request 03#479: Remove wiki.teessidehackspace.org.uk - 13https://github.com/miraheze/ssl/pull/479 [07:51:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/7e72a5740091...88005c7f28cb [07:51:51] [02miraheze/ssl] 07Universal-Omega 0388005c7 - Remove wiki.teessidehackspace.org.uk (#479) [07:51:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.88, 9.47, 9.85 [07:52:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.73, 9.35, 9.30 [07:53:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=funkeywiki --missing (END - exit=0) [07:53:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:45] !log restarted polandballwiki script with MH maint script account (after creating locally via script) [07:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:54:14] [02ssl] 07Reception123 closed pull request 03#461: Redirect test3 to betaheze - 13https://github.com/miraheze/ssl/pull/461 [07:54:16] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/88005c7f28cb...a64f59c884a3 [07:54:17] [02miraheze/ssl] 07Universal-Omega 03a64f59c - Redirect test3 to betaheze (#461) [07:55:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 9.93, 9.14 [07:56:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.52, 7.82, 7.60 [07:57:25] Reception123: did you specify -u Miraheze maintenance script for deletebatch.php? [07:57:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:57:41] dmehus: please see log above [07:57:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.47, 11.48, 10.51 [07:57:56] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:57:57] CosmicAlpha, I don't see the username specified? [07:58:06]