[00:00:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:03:12] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 304 failures. Last run 2 minutes ago with 304 failures. Failed resources (up to 3 shown): File[wikidiffs.ga_private],File[factorio-mods.tk],File[factorio-mods.tk_private],File[bluearchive.wiki] [00:06:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.98, 3.26 [00:06:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.32, 2.88 [00:08:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.21, 2.90 [00:10:14] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [00:11:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.42, 4.21, 3.37 [00:15:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:16:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.53, 8.49, 8.02 [00:17:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.37, 3.46, 3.35 [00:18:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.03, 3.86, 3.82 [00:18:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.66, 8.59, 8.10 [00:18:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.75, 8.17, 7.98 [00:19:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.20, 3.26 [00:22:27] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.57, 1.22 [00:24:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 2.80, 3.38 [00:24:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.39, 1.20 [00:26:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.10, 8.11, 8.13 [00:31:09] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [00:32:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.11, 8.46 [00:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 9.30, 8.62 [00:34:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.71, 1.34 [00:36:01] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.91, 8.10, 8.27 [00:36:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 4.68, 3.68 [00:36:14] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [00:36:14] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:36:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.88, 3.04 [00:36:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.57, 1.25, 1.22 [00:36:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.45, 4.29, 3.30 [00:37:07] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:11] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:37] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:43] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:39:06] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [00:39:06] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [00:39:35] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.339 second response time [00:39:38] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 1.016 second response time [00:39:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.28, 9.12, 8.66 [00:40:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:40:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.74, 3.33 [00:42:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.30, 3.47, 3.22 [00:42:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.06, 9.62, 8.68 [00:42:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 3.29, 3.21 [00:43:57] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.32, 9.74, 9.01 [00:44:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.34, 3.57 [00:44:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.44, 8.98, 8.57 [00:44:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.32, 2.70, 2.97 [00:47:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.80, 9.94, 9.29 [00:49:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.74, 9.37, 8.17 [00:50:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.76, 3.26 [00:50:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:51:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.85, 8.71, 8.07 [00:55:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.58, 8.10, 8.01 [00:57:51] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.54, 7.39, 8.23 [00:59:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.11, 8.09, 8.40 [01:09:00] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [01:09:00] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403825998306274 seconds [01:10:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:12:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.80, 8.83, 8.54 [01:12:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.60, 10.28, 9.00 [01:13:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:19:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.73, 8.23, 8.44 [01:20:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.88, 9.53, 9.31 [01:21:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.00, 8.81, 7.90 [01:23:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.89, 8.97, 8.73 [01:24:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.90, 10.04, 9.55 [01:25:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.53, 8.14, 7.75 [01:25:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.31, 8.13, 7.85 [01:27:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.69, 7.75, 7.64 [01:27:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.14, 8.11, 7.57 [01:28:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 9.65, 9.53 [01:28:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.29, 8.80, 7.75 [01:29:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 7.91, 7.55 [01:30:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.89, 10.37, 9.80 [01:32:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 8.78, 8.00 [01:34:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 10.00, 9.82 [01:34:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.66, 8.37, 7.94 [01:35:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.60, 2.97 [01:35:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [01:36:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.24, 10.30, 9.94 [01:37:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 3.39, 2.98 [01:40:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 8.35, 7.85 [01:40:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.08, 9.63, 9.83 [01:42:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.22, 8.17, 7.84 [01:45:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 8.33, 7.93 [01:47:50] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.48, 8.02, 7.87 [01:49:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.75, 9.54, 9.12 [01:51:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.41, 9.23, 9.07 [01:52:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.02, 9.29, 9.28 [01:53:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.43, 9.77, 9.29 [01:56:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.97, 9.81, 9.50 [01:57:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.19, 9.50, 9.31 [02:00:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 9.87, 9.57 [02:02:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.47, 9.32, 9.40 [02:06:12] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [02:08:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:09:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.52, 3.01 [02:09:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.04, 9.40, 9.21 [02:11:52] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.79, 1.22 [02:12:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:13:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.36, 3.08 [02:13:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.43, 1.15 [02:15:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.52, 9.12, 9.29 [02:21:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.04, 9.22, 9.24 [02:22:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:23:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.02, 8.60, 9.00 [02:27:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:30:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.08, 9.09, 8.85 [02:32:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.96, 9.51, 9.07 [02:32:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.61, 3.25 [02:34:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 9.95, 9.30 [02:34:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.35, 3.19 [02:34:49] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:34:49] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405406475067139 seconds [02:36:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.19, 9.90, 9.38 [02:36:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.20, 8.78, 8.17 [02:41:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.02, 9.40, 8.86 [02:42:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.63, 8.46, 8.28 [02:43:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.65, 9.00, 8.75 [02:44:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 0.95, 2.19, 1.60 [02:46:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.66, 1.72, 1.50 [02:47:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:48:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.15, 1.57, 1.48 [02:51:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.54, 2.64, 1.90 [02:53:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.95, 7.90, 8.46 [02:55:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.10, 1.81, 1.73 [02:57:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:57:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.53, 1.63 [02:58:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.76, 8.40, 8.08 [02:59:51] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.70, 9.84, 9.30 [03:00:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.39, 8.33, 8.09 [03:00:53] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [03:00:55] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818275192 [03:06:50] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [03:06:50] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:09:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.45, 9.92, 9.68 [03:10:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.74, 8.27, 7.58 [03:11:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.93, 8.49, 8.20 [03:11:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.53, 9.84, 9.66 [03:12:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.78, 7.90, 7.54 [03:13:19] miraheze/CreateWiki - lens0021 the build passed. [03:13:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:13:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.52, 9.94, 9.74 [03:15:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.46, 8.15, 8.18 [03:18:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:19:00] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818281486 [03:20:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.35, 8.30, 7.87 [03:21:41] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818282353 [03:22:09] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818282526 [03:23:25] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [03:23:33] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818283031 [03:23:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.48, 8.39, 8.01 [03:24:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 8.04, 7.55 [03:25:11] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818283593 [03:25:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.14, 9.30, 9.18 [03:25:58] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818283840 [03:26:02] miraheze/CreateWiki - lens0021 the build has errored. [03:26:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.86, 7.98, 7.59 [03:26:20] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818283964 [03:26:25] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818283994 [03:27:07] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818283031 [03:27:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.41, 8.58, 8.93 [03:31:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.06, 9.85, 9.33 [03:31:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [03:33:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.87, 9.10, 9.12 [03:33:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:47] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [03:36:22] miraheze/CreateWiki - lens0021 the build has errored. [03:38:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:39:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 4.04, 3.10 [03:39:32] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.44, 7.51, 8.35 [03:40:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.63, 2.99 [03:41:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.70, 3.08 [03:41:46] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818289185 [03:43:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:17] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [03:44:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 2.98, 2.93 [03:46:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.44, 2.74, 2.84 [03:46:53] miraheze/CreateWiki - lens0021 the build has errored. [03:49:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:52:10] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [03:54:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:56:57] miraheze/CreateWiki - lens0021 the build has errored. [03:58:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 8.66, 8.32 [03:58:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:01:17] [02CreateWiki] 07lens0021 commented on pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310#issuecomment-1057642158 [04:03:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:06:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.26, 7.84, 8.21 [04:13:59] [02CreateWiki] 07lens0021 synchronize pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [04:16:56] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818300743 [04:21:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:25:51] miraheze/CreateWiki - lens0021 the build passed. [04:26:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:36:01] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:51:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:56:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:58:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [04:58:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407238721847534 seconds [05:10:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 8.99, 7.64 [05:12:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.80, 8.51, 7.63 [05:13:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.85, 8.06, 7.27 [05:14:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.87, 8.11, 7.61 [05:15:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.84, 7.82, 7.28 [05:21:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.11, 8.55, 7.71 [05:23:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.48, 7.74, 7.52 [05:27:56] PROBLEM - clockupwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'clockupwiki.org' expires in 15 day(s) (Sat 19 Mar 2022 05:21:52 GMT +0000). [05:29:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/74f978927054...260c72657ee3 [05:29:40] [02miraheze/ssl] 07MirahezeSSLBot 03260c726 - Bot: Update SSL cert for clockupwiki.org [05:37:07] PROBLEM - wiki.meower.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meower.org' expires in 15 day(s) (Sat 19 Mar 2022 05:17:04 GMT +0000). [05:37:27] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [05:37:27] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:39:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/260c72657ee3...06e1ebfac5dd [05:39:05] [02miraheze/ssl] 07MirahezeSSLBot 0306e1ebf - Bot: Update SSL cert for wiki.meower.org [05:56:51] RECOVERY - clockupwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'clockupwiki.org' will expire on Wed 01 Jun 2022 04:29:33 GMT +0000. [05:58:35] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [06:04:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:06:13] RECOVERY - wiki.meower.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meower.org' will expire on Wed 01 Jun 2022 04:38:56 GMT +0000. [06:07:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:09:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:27:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:40:04] [02mw-config] 07Reception123 closed pull request 03#4472: disable sidebarcache for solarawiki per T8732 - 13https://github.com/miraheze/mw-config/pull/4472 [06:40:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5d3452c6dcb3...93f8d0c30195 [06:40:07] [02miraheze/mw-config] 07Reception123 0393f8d0c - disable sidebarcache for solarawiki per T8732 (#4472) [06:40:08] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [06:40:10] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [06:41:18] miraheze/mw-config - Reception123 the build passed. [06:49:01] !log [@test101] starting deploy of {'config': True} to all [06:49:02] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [06:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:49:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:49:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9014fb6b0fe4 [06:49:35] [02miraheze/mw-config] 07Reception123 039014fb6 - add ArticlePlaceholder settings to ManageWiki per T8818 [06:49:36] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [06:49:38] [02mw-config] 07Reception123 opened pull request 03#4487: add ArticlePlaceholder settings to ManageWiki per T8818 - 13https://github.com/miraheze/mw-config/pull/4487 [06:50:47] miraheze/mw-config - Reception123 the build passed. [06:52:28] !log [@mwtask111] starting deploy of {'config': True} to all [06:52:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:40] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [06:52:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:39] [02puppet] 07Reception123 commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057733450 [07:07:45] [02puppet] 07dmehus commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057741053 [07:09:12] [02puppet] 07Reception123 commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057741883 [07:10:56] [02puppet] 07dmehus commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057742969 [07:15:31] [02puppet] 07Universal-Omega commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057745583 [07:16:15] [02puppet] 07Universal-Omega commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057746036 [07:28:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [07:28:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4065327644348145 seconds [07:32:57] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818382489 [07:34:05] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818383121 [07:38:56] [02CreateWiki] 07RhinosF1 commented on pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310#issuecomment-1057759175 [07:53:23] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818394790 [07:59:29] [02puppet] 07JohnFLewis commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1057775521 [07:59:55] [02CreateWiki] 07lens0021 reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818394790 [08:00:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:05:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:13:02] JohnLewis: that bad gateway error is very strange, any idea where else I could look? [08:13:03] [tell] Reception123: 2022-03-03 - 02:56:51CET tell Reception123 Sent you a MemoServ memo re: your ping in -ops [08:14:01] Reception123: php logs, MediaWiki logs might provide an insight, otherwise you have grafana for things like php execution times, capacity overview etc. [08:14:41] hmm, ok [08:18:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.41, 1.21 [08:20:38] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.12, 1.27, 1.18 [08:28:33] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [08:32:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.30, 8.80, 7.64 [08:34:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.59, 8.01, 7.50 [09:01:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.82, 8.40, 7.70 [09:03:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.16, 8.16, 7.71 [09:03:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:07:18] [02CreateWiki] 07Universal-Omega reviewed pull request 03#310 commit - 13https://github.com/miraheze/CreateWiki/pull/310#discussion_r818449554 [09:08:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:09:51] [02puppet] 07Reception123 closed pull request 03#2445: Bump actions - 13https://github.com/miraheze/puppet/pull/2445 [09:09:53] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c14fdd177588...49cae26a2243 [09:09:54] [02miraheze/puppet] 07Universal-Omega 0349cae26 - Bump actions (#2445) [09:09:56] [02phabricator-extensions] 07Reception123 closed pull request 03#20: Bump actions/checkout from 2 to 3 - 13https://github.com/miraheze/phabricator-extensions/pull/20 [09:09:57] [02miraheze/phabricator-extensions] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/032e738d2106...3c9fe0f67ee7 [09:09:59] [02miraheze/phabricator-extensions] 07Universal-Omega 033c9fe0f - Bump actions/checkout from 2 to 3 (#20) [09:11:10] miraheze/phabricator-extensions - Reception123 the build passed. [10:08:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:13:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:26:29] PROBLEM - betaheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.betaheze.org' expires in 15 day(s) (Sat 19 Mar 2022 10:24:21 GMT +0000). [10:28:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [10:28:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407589435577393 seconds [10:32:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/06e1ebfac5dd...e0f605c9eac6 [10:32:37] [02miraheze/ssl] 07MirahezeSSLBot 03e0f605c - Bot: Update SSL cert for betaheze.org [10:44:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [10:44:30] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:58:33] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [11:23:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:28:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:28:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407288312911987 seconds [11:28:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:31:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.05, 8.86, 7.22 [11:34:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 8.77, 7.39 [11:36:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.24, 8.47, 7.44 [11:36:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:43:57] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:00:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:05:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:13:10] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.95, 1.51, 0.94 [12:17:06] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.24, 1.82, 1.19 [12:19:05] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.59, 1.69, 1.21 [12:24:57] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.92, 1.84, 1.42 [12:26:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.49, 1.69, 1.42 [12:29:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 8.13, 7.34 [12:31:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.56, 7.65, 7.27 [12:48:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.59, 2.72 [12:48:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.35, 0.72 [12:50:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 3.15, 2.67 [12:50:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.12, 0.71 [12:52:37] [02mediawiki] 07RhinosF1 opened pull request 03#5655: Update with changes from wikimedia:mediawiki - 13https://github.com/miraheze/mediawiki/pull/5655 [12:53:26] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.08, 1.76, 1.53 [12:55:24] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.50, 1.63, 1.51 [12:58:33] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [12:59:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:04:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:19:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:23:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.56, 8.66, 7.77 [13:24:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:28:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4071619510650635 seconds [13:43:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.06, 8.08, 8.40 [13:47:45] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:47:45] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:51:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:56:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:58:36] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [13:59:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:01:59] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.85, 10.41, 9.13 [14:03:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.81, 9.64, 9.02 [14:04:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:15:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:15:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.48, 8.01, 8.42 [14:17:29] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [14:19:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.61, 8.62, 7.57 [14:21:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 8.50, 7.65 [14:22:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.99, 8.54, 8.02 [14:23:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.13, 7.87, 7.52 [14:24:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.40, 8.12, 7.93 [14:25:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.21, 1.56, 1.07 [14:27:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.72, 9.39, 8.22 [14:27:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.31, 1.48, 1.10 [14:28:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [14:28:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407053709030151 seconds [14:29:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.61, 9.44, 8.39 [14:30:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.60, 10.12, 8.87 [14:31:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.09, 9.64, 8.58 [14:31:31] [02mediawiki] 07wmfgerrit synchronize pull request 03#5655: Update with changes from wikimedia:mediawiki - 13https://github.com/miraheze/mediawiki/pull/5655 [14:31:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.98, 8.63, 8.22 [14:32:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.72, 9.44, 8.80 [14:33:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.16, 9.13, 8.51 [14:33:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.37, 7.82, 7.97 [14:34:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.03, 7.79, 8.27 [14:34:56] [02mediawiki] 07wmfgerrit synchronize pull request 03#5655: Update with changes from wikimedia:mediawiki - 13https://github.com/miraheze/mediawiki/pull/5655 [14:35:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.30, 8.35, 8.31 [14:35:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:36:10] [02mediawiki] 07wmfgerrit synchronize pull request 03#5655: Update with changes from wikimedia:mediawiki - 13https://github.com/miraheze/mediawiki/pull/5655 [14:42:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:31] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.07, 9.45, 8.43 [14:45:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.30, 8.72, 8.30 [14:47:13] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:47:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.08, 8.21, 8.17 [14:49:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.64, 8.71, 8.35 [14:51:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.28, 8.28, 8.24 [14:52:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:53:07] [02mediawiki] 07wmfgerrit synchronize pull request 03#5655: Update with changes from wikimedia:mediawiki - 13https://github.com/miraheze/mediawiki/pull/5655 [14:54:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.58, 8.45, 8.11 [14:57:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/49cae26a2243...8d942b5df6a4 [14:57:05] [02miraheze/puppet] 07paladox 038d942b5 - mon111: Increase MaxConcurrentChecks to 40 [14:57:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 8.77, 8.37 [14:57:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:58:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.33, 8.47, 8.21 [14:59:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.24, 8.17, 8.20 [15:02:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 8.98, 8.52 [15:02:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:03:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.84, 8.58, 8.38 [15:05:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.44, 8.11, 8.23 [15:06:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.69, 8.00, 8.24 [15:09:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:12:19] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.16, 8.59 [15:14:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.81, 8.70, 8.51 [15:16:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.48, 8.30, 8.38 [15:17:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.03, 8.50, 7.50 [15:18:54] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in