[00:02:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [00:02:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:03:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 0.98, 2.27, 1.43 [00:04:35] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 700 failures. Last run 3 minutes ago with 700 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:05:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.41, 1.63, 1.30 [00:06:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 8.15, 7.01 [00:06:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.28, 3.06 [00:08:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.17, 7.25, 6.81 [00:08:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.64, 2.67, 2.86 [00:12:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.41, 3.14 [00:14:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.77, 3.35 [00:14:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.08, 3.05 [00:16:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.49, 3.30 [00:20:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.02, 2.98, 3.15 [00:20:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:25:37] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:31:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:32:31] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:40:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.44, 2.97 [00:41:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.07, 8.98, 7.47 [00:41:45] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user MacFan4000 (END - exit=0) [00:41:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.58, 8.21, 7.37 [00:45:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.01, 4.64, 3.39 [00:47:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.05, 8.54, 8.11 [00:50:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.64, 3.07 [00:51:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.30, 8.42, 8.16 [00:52:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 3.19, 2.98 [00:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:59:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.69, 3.73 [01:00:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:02:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.87, 3.97 [01:04:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.22, 4.08 [01:07:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.29, 3.10, 3.38 [01:08:18] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/1d6bebdcb119 [01:08:19] [02miraheze/ssl] 07MacFan4000 031d6bebd - Create wiki.dereric.gq.crt [01:08:21] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [01:08:29] [02ssl] 07MacFan4000 opened pull request 03#483: add wiki.dereric.gq cert - 13https://github.com/miraheze/ssl/pull/483 [01:09:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:10:24] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1d6bebdcb119...ee341f4dd395 [01:10:26] [02miraheze/ssl] 07MacFan4000 03ee341f4 - Update certs.yaml [01:10:27] [02ssl] 07MacFan4000 synchronize pull request 03#483: add wiki.dereric.gq cert - 13https://github.com/miraheze/ssl/pull/483 [01:10:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.60, 3.89 [01:10:45] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/71670dec653c...df9d2c29c27b [01:10:47] [02miraheze/ssl] 07MacFan4000 03df9d2c2 - add wiki.dereric.gq cert (#483) [01:10:48] [02ssl] 07MacFan4000 closed pull request 03#483: add wiki.dereric.gq cert - 13https://github.com/miraheze/ssl/pull/483 [01:10:50] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [01:10:51] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [01:14:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:16:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.81, 2.68, 3.40 [01:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:26:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.75, 1.22 [01:30:53] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.37, 0.97, 1.02 [01:34:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:40:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.58, 1.24 [01:42:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.67, 1.49, 1.24 [01:46:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.64, 3.26 [01:48:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 3.19, 3.15 [01:53:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:53:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.09, 8.34, 7.63 [01:54:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.56, 7.78, 6.82 [01:56:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 8.05, 6.96 [01:56:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.34, 3.12 [01:56:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.11, 7.60, 6.88 [01:57:44] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.19, 8.07, 7.70 [01:58:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.37, 3.28, 3.12 [02:00:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.46, 7.20, 6.91 [02:06:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.46, 3.10 [02:08:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.16, 3.40 [02:09:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 3.82, 3.32 [02:10:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.79, 3.35 [02:11:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.44, 3.24 [02:12:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 8.49, 7.91 [02:13:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 2.92, 3.07 [02:14:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.66, 3.36 [02:14:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 7.94, 7.78 [02:17:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.70, 1.72, 1.18 [02:18:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:19:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.50, 9.07, 8.06 [02:19:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.28, 1.08 [02:20:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.86, 3.59 [02:21:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.24, 9.29, 8.28 [02:23:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.75, 9.72, 8.55 [02:24:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 4.14, 3.76 [02:25:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.87, 8.39, 8.22 [02:26:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.66, 3.63 [02:28:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 10.75, 5.44, 4.22 [02:29:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.60, 3.16 [02:31:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.53, 3.05 [02:33:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.36, 3.17 [02:35:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 3.02, 2.97 [02:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:40:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.65, 3.94 [02:41:16] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.29, 8.96, 8.25 [02:42:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.53, 8.13, 7.98 [02:42:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 8.21, 7.67 [02:43:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 8.99, 8.36 [02:44:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.46, 8.69, 7.96 [02:44:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 7.93, 7.64 [02:46:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.03, 8.05, 7.82 [02:46:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.50, 9.33, 8.46 [02:47:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.28, 9.18, 8.52 [02:49:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.76, 8.79, 8.45 [02:50:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 2.68, 3.34 [02:50:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 9.94, 8.95 [02:51:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.10, 9.16, 8.61 [02:52:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.01, 10.31, 9.20 [02:53:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.13, 9.06, 8.64 [02:54:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 9.37, 8.97 [02:57:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.43, 8.15, 8.36 [02:59:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.37, 2.89 [02:59:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.55, 3.11 [02:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.96, 7.93, 8.43 [03:01:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.23, 3.05 [03:02:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.64, 3.15 [03:04:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.30, 3.08 [03:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:10:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 8.30, 8.25 [03:10:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.74, 8.51, 7.84 [03:12:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.25, 7.80, 7.68 [03:13:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.97, 8.04, 8.14 [03:16:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:17:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.69, 9.49, 8.64 [03:19:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.84, 8.73, 8.46 [03:19:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.23, 8.72, 8.39 [03:21:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.33, 7.90, 8.19 [03:25:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.57, 7.74, 8.16 [03:43:20] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.09, 9.16, 8.12 [03:45:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.43, 8.19, 7.89 [03:48:54] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [03:48:54] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40518593788147 seconds [03:50:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 8.53, 8.03 [03:53:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:56:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.88, 7.69, 7.91 [03:58:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:17:45] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [04:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.25, 2.75 [04:26:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.06, 3.26, 2.68 [04:26:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.31, 3.38, 2.87 [04:28:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.33, 2.79 [04:30:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.64, 2.77, 2.66 [04:30:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.41, 3.01 [04:32:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 3.32, 3.03 [04:51:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.70, 9.45, 8.04 [04:51:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.26, 9.85, 8.10 [04:53:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.76, 9.07, 8.09 [04:53:20] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.50, 12.06, 9.40 [04:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:53:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.62, 9.20, 8.10 [04:55:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.46, 8.32, 7.95 [04:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:59:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.72, 9.48, 9.16 [04:59:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.89, 8.31, 8.05 [05:00:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.51, 3.16 [05:03:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.22, 7.61, 8.44 [05:04:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.25, 3.13 [05:12:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 7.92, 7.08 [05:13:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.85, 7.48, 7.02 [05:14:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.81, 8.25, 7.99 [05:15:08] [02ManageWiki] 07Universal-Omega commented on pull request 03#337: Convert to PSR4 - 13https://github.com/miraheze/ManageWiki/pull/337#issuecomment-1053151763 [05:15:32] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [05:15:32] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406681776046753 seconds [05:16:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.87, 7.84, 7.88 [05:26:57] [02ManageWiki] 07lens0021 synchronize pull request 03#337: Convert to PSR4 - 13https://github.com/miraheze/ManageWiki/pull/337 [05:30:00] miraheze/ManageWiki - lens0021 the build has errored. [05:32:21] [02ManageWiki] 07lens0021 synchronize pull request 03#337: Convert to PSR4 - 13https://github.com/miraheze/ManageWiki/pull/337 [05:35:54] miraheze/ManageWiki - lens0021 the build has errored. [05:36:02] [02ManageWiki] 07lens0021 synchronize pull request 03#337: Convert to PSR4 - 13https://github.com/miraheze/ManageWiki/pull/337 [05:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 8.83, 7.81 [05:43:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.82, 8.03, 7.64 [05:44:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [05:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:13] miraheze/ManageWiki - lens0021 the build passed. [06:00:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 8.42, 7.63 [06:02:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.64, 8.32, 7.69 [06:06:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 7.87, 7.68 [06:10:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.31, 2.83 [06:12:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.12, 2.82 [06:28:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 3.68, 2.62 [06:30:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.19, 2.57 [06:30:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:42:13] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [06:42:13] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406617164611816 seconds [06:44:32] [02ManageWiki] 07Universal-Omega closed pull request 03#337: Convert to PSR4 - 13https://github.com/miraheze/ManageWiki/pull/337 [06:44:33] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+20/-20/±12] 13https://github.com/miraheze/ManageWiki/compare/d8022ca8272f...4635bbf2a8f3 [06:44:35] [02miraheze/ManageWiki] 07lens0021 034635bbf - Convert to PSR4 (#337) [06:50:13] miraheze/ManageWiki - Universal-Omega the build passed. [06:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:55:06] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.89, 1.70, 1.43 [06:57:06] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.45, 1.58, 1.42 [07:01:06] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.87, 1.76, 1.53 [07:03:06] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.46, 1.66, 1.53 [07:04:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.43, 2.92 [07:06:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 2.79, 2.74 [07:09:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.71, 1.16 [07:11:05] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.89, 1.36, 1.09 [07:11:05] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:19:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:27:15] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [07:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:56:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.41, 2.92 [07:57:49] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:58:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.03, 2.84 [08:26:27] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [08:33:00] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [08:33:01] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:57:01] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:02:35] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:32:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:32:09] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:37:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [09:37:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.409748315811157 seconds [10:00:23] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/commit/2244e11e9ef1 [10:00:24] [02miraheze/CreateWiki] 07Universal-Omega 032244e11 - Add RemoteWiki phpunit tests [10:00:26] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [10:00:27] [02CreateWiki] 07Universal-Omega opened pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [10:01:43] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:02:58] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [10:03:00] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/2244e11e9ef1...b7bb0f1a6128 [10:03:01] [02miraheze/CreateWiki] 07Universal-Omega 03b7bb0f1 - Update RemoteWikiTest.php [10:03:09] miraheze/CreateWiki - Universal-Omega the build has errored. [10:06:47] miraheze/CreateWiki - Universal-Omega the build has errored. [10:07:47] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b7bb0f1a6128...d5567633b349 [10:07:49] [02miraheze/CreateWiki] 07Universal-Omega 03d556763 - Update RemoteWikiTest.php [10:07:50] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [10:07:52] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [10:11:11] miraheze/CreateWiki - Universal-Omega the build has errored. [10:13:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d5567633b349...88e3b44d2005 [10:13:10] [02miraheze/CreateWiki] 07Universal-Omega 0388e3b44 - Update RemoteWikiTest.php [10:13:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [10:18:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:18:45] miraheze/CreateWiki - Universal-Omega the build passed. [10:22:15] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/88e3b44d2005...30944eac5d0e [10:22:17] [02miraheze/CreateWiki] 07Universal-Omega 0330944ea - Update RemoteWikiTest.php [10:22:18] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [10:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:25:43] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/30944eac5d0e...bcbcd1c94c65 [10:25:44] [02miraheze/CreateWiki] 07Universal-Omega 03bcbcd1c - Update RemoteWikiTest.php [10:25:46] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [10:26:54] miraheze/CreateWiki - Universal-Omega the build has errored. [10:33:18] miraheze/CreateWiki - Universal-Omega the build passed. [10:44:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.36, 3.58, 2.75 [10:46:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.32, 2.74 [10:48:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.05, 2.73 [10:49:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.27, 9.30, 7.59 [10:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:51:52] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.80, 8.04, 7.35 [10:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:03:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.24, 2.69 [11:05:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:05:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 3.24, 2.76 [11:07:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:07:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407036304473877 seconds [11:34:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:37:52] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [11:56:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.71, 2.92 [11:58:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 3.14, 2.80 [12:01:35] PROBLEM - wiki.pacebi.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.23438744057152:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1543:SSL alert number 40 [12:03:54] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [12:03:54] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:13:06] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.75, 1.45, 0.92 [12:17:06] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.65, 1.60, 1.11 [12:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:23:06] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.81, 1.34 [12:25:06] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.44, 1.67, 1.35 [12:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:26:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.85, 3.34 [12:28:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.46, 2.82 [12:29:14] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.71, 1.92, 1.28 [12:30:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.94, 3.77, 2.95 [12:30:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.67, 2.98 [12:31:10] RECOVERY - wiki.pacebi.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pacebi.org' will expire on Sun 15 May 2022 07:56:38 GMT +0000. [12:31:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.05, 1.61, 1.24 [12:32:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.22, 2.85 [12:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:32:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 3.08, 2.85 [12:34:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.71, 3.30, 3.47 [12:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:38:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.07, 3.34 [12:41:26] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.90, 1.76, 1.56 [12:43:20] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.47, 1.65, 1.55 [12:46:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.77, 3.49 [12:48:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.59, 3.45 [12:50:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.67, 3.48 [12:53:06] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.33, 1.84, 1.64 [12:57:06] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.86, 1.91, 1.73 [12:59:06] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.42, 1.69, 1.67 [13:02:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.61, 3.77 [13:07:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:07:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4059364795684814 seconds [13:08:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.45, 3.61 [13:12:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.44, 3.57 [13:14:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 3.77, 3.66 [13:16:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.62, 3.63 [13:20:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.61, 2.96, 3.35 [13:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:37:31] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 10.51, 6.63, 3.77 [13:37:52] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [13:41:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.58, 10.28, 8.25 [13:43:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.04, 9.73, 8.32 [13:43:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:47:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.90, 8.42, 8.05 [13:48:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:51:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.80, 9.37, 8.47 [13:55:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.72, 9.99, 8.99 [13:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:07:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [14:07:56] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405636310577393 seconds [14:07:56] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=tolololpediawik --wiki (END - exit=256) [14:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:21] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=tolololpediawik (END - exit=65280) [14:08:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:08:35] !log [macfan@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=tolololpediawiki (END - exit=0) [14:08:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:14:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.22, 9.84, 9.44 [14:16:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 9.29, 9.29 [14:16:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.29, 8.67, 7.84 [14:18:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 10.43, 9.70 [14:18:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.14, 8.13, 7.75 [14:20:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.36, 9.80, 9.57 [14:22:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 4.53, 3.56 [14:24:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.75, 3.38 [14:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:26:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 3.36, 3.28 [14:32:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 9.42, 9.27 [14:33:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [14:33:43] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:37:52] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [14:39:14] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.85, 1.56, 1.22 [14:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:39:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 8.94, 8.07 [14:40:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 9.79, 9.64 [14:40:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.43, 3.32 [14:41:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.21, 1.38, 1.19 [14:41:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.51, 7.22, 7.54 [14:44:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.36, 3.34 [14:48:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.15, 9.06, 9.03 [14:51:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.97, 8.68, 8.97 [14:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:57:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.14, 4.66, 3.30 [15:01:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.89, 3.29 [15:01:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.98, 9.53, 9.16 [15:03:18] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:03:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.43, 8.64, 8.88 [15:07:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.26, 3.20 [15:07:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:09:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.31, 9.32, 9.06 [15:10:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.96, 8.58, 7.36 [15:11:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.83, 3.46 [15:12:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 8.14, 7.35 [15:13:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.80, 9.06, 9.03 [15:15:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.98, 3.03, 3.23 [15:17:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:17:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.82, 9.61, 9.19 [15:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:26:02] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.71, 1.35 [15:27:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.54, 9.78, 9.83 [15:27:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.66, 1.31, 1.24 [15:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:29:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 9.78, 10.14, 9.97 [15:31:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 8.14, 7.18 [15:31:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.22, 9.34, 9.70 [15:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:33:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.80, 8.17, 7.32 [15:34:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.11, 2.86 [15:35:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.35, 9.56, 9.68 [15:36:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 2.99, 2.85 [15:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:37:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.41, 8.84, 9.41 [15:39:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.14, 2.96 [15:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:41:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.37, 9.48 [15:43:38]