[00:00:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.57, 3.53 [00:00:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.15, 9.79, 9.37 [00:02:16] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7142ab66ee95...467dad000d22 [00:02:17] [02miraheze/CreateWiki] 07Universal-Omega 03467dad0 - Update WikiManagerTest.php [00:02:19] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:02:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 9.31, 9.24 [00:04:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.66, 3.55 [00:04:33] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 201 failures. Last run 3 minutes ago with 201 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:04:58] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:05:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 10.21, 9.11 [00:05:31] miraheze/CreateWiki - Universal-Omega the build has errored. [00:06:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.43, 3.48 [00:06:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 3.92, 3.48 [00:09:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.41, 3.11 [00:09:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.30, 9.29, 9.28 [00:10:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.54, 3.51 [00:11:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.30, 3.10 [00:12:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.47, 3.50 [00:12:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.54, 9.26, 9.25 [00:14:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.75, 3.59 [00:15:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 9.54, 9.32 [00:15:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 4.08, 3.47 [00:16:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.60, 3.58 [00:16:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.27, 9.22 [00:17:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.82, 3.43 [00:17:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.62, 9.21, 9.23 [00:18:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.79, 3.05, 3.38 [00:18:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.13, 8.85, 9.08 [00:18:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.52, 7.79, 8.37 [00:19:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.94, 3.17, 3.24 [00:19:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.07, 9.89, 9.89 [00:20:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.05, 9.61, 9.35 [00:21:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.58, 9.11, 9.91 [00:21:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.52, 10.03, 9.93 [00:22:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.68, 3.89 [00:22:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.55, 9.36 [00:23:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.55, 9.96 [00:23:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.85, 9.51, 9.75 [00:24:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.30, 10.29, 9.66 [00:24:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.79, 3.89 [00:25:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 9.44, 9.87 [00:26:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.76, 3.88 [00:26:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.24, 8.52, 8.41 [00:27:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.80, 3.47 [00:27:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.95, 10.01 [00:28:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.32, 9.33, 9.45 [00:28:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.76, 7.74, 8.14 [00:29:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.33, 9.79, 9.95 [00:29:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.53, 9.42, 9.13 [00:30:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.83, 3.87 [00:30:54] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/467dad000d22...e80abc39af78 [00:30:55] [02miraheze/CreateWiki] 07Universal-Omega 03e80abc3 - Update WikiManagerTest.php [00:30:57] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:31:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.78, 3.55 [00:31:11] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e80abc39af78...42b66f306491 [00:31:13] [02miraheze/CreateWiki] 07Universal-Omega 0342b66f3 - Update WikiManagerTest.php [00:31:14] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:31:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.10, 8.81, 8.94 [00:32:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.61, 3.40 [00:32:31] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:42] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/42b66f306491...d98e20ac95c2 [00:32:44] [02miraheze/CreateWiki] 07Universal-Omega 03d98e20a - Update WikiManagerTest.php [00:32:45] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:33:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.57, 10.11, 9.95 [00:34:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.77, 9.43, 9.43 [00:35:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 9.74, 9.85 [00:35:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.71, 9.33, 9.44 [00:36:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.07, 4.51, 3.76 [00:37:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.99, 9.04, 9.34 [00:38:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.69, 9.10, 9.41 [00:38:31] miraheze/CreateWiki - Universal-Omega the build passed. [00:39:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.03, 4.93, 4.01 [00:39:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 7.72, 8.39 [00:40:02] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:32] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:35] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:47] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:49] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:51] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.59, 8.17, 9.56 [00:41:00] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:00] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:00] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:09] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:13] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:22] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:41:59] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 0.614 second response time [00:42:17] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d98e20ac95c2...3188fab89812 [00:42:18] [02miraheze/CreateWiki] 07Universal-Omega 033188fab - Update WikiManagerTest.php [00:42:20] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:42:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.06, 3.98, 3.87 [00:42:27] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [00:42:35] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.330 second response time [00:42:43] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 1.175 second response time [00:42:46] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.335 second response time [00:42:47] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.016 second response time [00:42:47] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 0.825 second response time [00:42:56] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.428 second response time [00:42:59] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 0.728 second response time [00:43:00] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.332 second response time [00:43:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.79, 3.87, 3.80 [00:43:07] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.900 second response time [00:43:08] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.316 second response time [00:43:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.25, 8.88, 9.27 [00:43:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.851 second response time [00:44:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.90, 9.67, 9.79 [00:45:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.63, 8.72, 9.19 [00:45:58] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3188fab89812...86d7ab794e2a [00:45:59] [02miraheze/CreateWiki] 07Universal-Omega 0386d7ab7 - Update WikiManagerTest.php [00:46:01] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:46:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.46, 3.14, 3.86 [00:46:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.99, 8.30, 9.26 [00:46:59] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.64, 2.59, 3.28 [00:47:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.90, 1.20, 1.97 [00:47:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.34, 7.77, 8.48 [00:48:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.22, 7.45, 8.41 [00:48:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 2.64, 3.29 [00:48:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/86d7ab794e2a...0b2ee82bb0cd [00:48:31] [02miraheze/CreateWiki] 07Universal-Omega 030b2ee82 - Update WikiManager.php [00:48:33] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:48:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.29, 9.25, 9.48 [00:48:57] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.331 second response time [00:49:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 9.16, 9.15 [00:50:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.84, 9.52, 9.56 [00:51:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 8.78, 9.01 [00:51:28] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0b2ee82bb0cd...ed8f647190ee [00:51:29] [02miraheze/CreateWiki] 07Universal-Omega 03ed8f647 - Update WikiManager.php [00:51:31] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [00:51:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.12, 1.00, 1.70 [00:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:54:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 2.68, 3.35 [00:55:04] miraheze/CreateWiki - Universal-Omega the build has errored. [00:56:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.63, 9.52 [00:57:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.42, 7.79, 8.49 [00:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:58:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.59, 8.91, 9.28 [01:02:48] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:02:48] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:04:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.30, 7.49, 8.50 [01:05:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.73, 8.31, 8.09 [01:07:51] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [01:07:52] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.41039252281189 seconds [01:08:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 8.10, 7.75 [01:09:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.35, 8.82, 8.30 [01:10:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 7.85, 7.70 [01:12:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.49, 3.19 [01:12:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:12:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.12, 2.79 [01:13:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.58, 8.96, 8.54 [01:13:41] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [01:14:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.14, 3.92, 3.37 [01:14:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 2.75, 2.70 [01:14:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.73, 8.72, 8.06 [01:15:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.91, 9.46, 8.75 [01:16:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.98, 8.80, 8.47 [01:18:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.32, 8.43 [01:18:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.24, 8.28, 8.32 [01:19:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.69, 8.93, 8.77 [01:19:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.65, 8.80, 8.66 [01:20:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.99, 3.65 [01:20:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.67, 9.01, 8.43 [01:21:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 9.46, 8.92 [01:22:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.33, 9.00, 8.60 [01:22:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.56, 9.55, 8.69 [01:24:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.83, 2.90, 3.28 [01:24:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.98, 8.42, 8.42 [01:24:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.00, 9.00, 8.60 [01:24:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.09, 7.64, 8.28 [01:28:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.90, 10.18, 8.88 [01:28:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.41, 8.38, 8.45 [01:29:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.09, 9.17, 9.23 [01:30:15] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ed8f647190ee...a4b2667447d6 [01:30:16] [02miraheze/CreateWiki] 07Universal-Omega 03a4b2667 - Update WikiManager.php [01:30:18] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:35:31] miraheze/CreateWiki - Universal-Omega the build passed. [01:36:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.29, 3.74 [01:36:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.79, 9.41, 9.04 [01:37:03] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.40, 2.75, 1.81 [01:37:14] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [01:38:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.76, 3.60 [01:38:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.64, 4.22, 3.31 [01:40:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.46, 4.15, 3.77 [01:41:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.68, 1.57 [01:41:02] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a4b2667447d6...6bad8a01e6a0 [01:41:04] [02miraheze/CreateWiki] 07Universal-Omega 036bad8a0 - Update WikiManager.php [01:41:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.58, 9.21, 9.02 [01:41:05] [02CreateWiki] 07Universal-Omega synchronize pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [01:42:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.05, 3.39, 3.53 [01:43:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 9.17, 9.03 [01:44:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.47, 3.29 [01:44:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.12, 9.03, 8.56 [01:45:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.09, 8.91, 8.40 [01:46:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.17, 3.40 [01:46:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.20, 3.22 [01:46:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.75, 8.91, 8.58 [01:46:47] miraheze/CreateWiki - Universal-Omega the build passed. [01:48:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 7.75, 8.35 [01:48:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.50, 8.49, 8.34 [01:50:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.46, 8.19, 8.38 [01:52:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.99, 8.90, 8.58 [01:53:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.69, 8.97, 8.12 [01:54:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 8.90, 8.65 [01:55:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.98, 9.20, 8.46 [01:56:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.45, 3.33 [01:57:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 9.35, 8.49 [02:00:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.74, 3.50 [02:00:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:02:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.33, 3.38 [02:03:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.26, 8.81, 8.48 [02:03:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.24, 10.26, 9.12 [02:04:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.16, 8.21, 8.46 [02:04:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.04, 8.17, 8.48 [02:05:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 9.60, 8.77 [02:05:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.58, 8.92, 8.77 [02:06:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:06:38] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406416177749634 seconds [02:07:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.87, 9.07, 8.67 [02:07:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.57, 8.29, 8.43 [02:08:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.28, 8.19, 8.48 [02:09:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.62, 8.02, 8.41 [02:11:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.84, 4.41, 3.76 [02:11:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.72, 8.26, 8.48 [02:11:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.50, 8.43, 8.35 [02:11:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.97, 9.80, 9.01 [02:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.62, 7.95, 8.20 [02:13:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.46, 9.51, 9.01 [02:15:39] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:19:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.70, 3.70 [02:19:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.84, 9.61, 8.89 [02:21:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.81, 3.73 [02:22:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.45, 4.11, 3.14 [02:23:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.64, 3.70 [02:23:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.21, 9.79, 9.19 [02:24:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.60, 3.08 [02:25:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.94, 9.54, 9.19 [02:25:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.34, 9.20, 9.04 [02:26:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.18, 2.99 [02:27:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 9.85, 9.31 [02:28:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 9.04, 8.49 [02:28:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.39, 8.82, 8.37 [02:29:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.69, 9.50, 9.26 [02:30:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.38, 9.75, 8.78 [02:32:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.83, 7.82, 8.17 [02:32:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.11, 9.35, 8.76 [02:32:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.75, 8.19, 8.26 [02:32:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 4.15, 3.85 [02:34:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.47, 3.63 [02:35:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.24, 9.55, 9.22 [02:35:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 9.08, 8.45 [02:36:00] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [02:36:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.14, 8.64 [02:36:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.19, 8.12, 8.40 [02:37:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.56, 8.82, 9.00 [02:37:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:38:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.39, 3.83, 3.70 [02:39:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.28, 9.16, 9.08 [02:39:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.65, 9.89, 8.89 [02:39:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.20, 9.77, 9.32 [02:40:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.11, 8.34, 8.41 [02:40:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.41, 3.56 [02:41:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.39, 9.19 [02:41:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 9.54, 8.89 [02:43:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 9.10, 8.71 [02:43:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 9.61, 9.40 [02:44:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.26, 10.16, 8.93 [02:44:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 2.93, 3.32 [02:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.92, 9.56, 9.25 [02:45:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 8.99, 8.73 [02:46:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.67, 8.61, 8.41 [02:48:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.23, 9.63, 9.05 [02:49:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 9.87, 9.20 [02:51:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.78, 9.63, 9.46 [02:51:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.95, 9.40, 9.02 [02:52:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.98, 3.58 [02:52:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.46, 4.28, 3.47 [02:53:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.86, 10.38, 9.56 [02:54:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 8.17, 8.35 [02:55:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 9.69, 9.40 [02:57:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.36, 9.37, 9.02 [02:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.65, 3.65 [02:58:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 4.00, 3.65 [02:59:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.38, 9.39, 9.08 [02:59:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.13, 9.44, 9.22 [03:00:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 4.02, 3.79 [03:00:38] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.87, 4.65, 3.92 [03:01:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.15, 9.94, 9.31 [03:01:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.26, 8.75, 9.00 [03:02:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [03:02:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:02:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.83, 3.76 [03:02:57] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.59, 7.49, 8.43 [03:03:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.81, 9.26, 9.15 [03:03:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.45, 9.34, 8.87 [03:05:23] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [03:05:23] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405519008636475 seconds [03:05:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.93, 8.57, 8.63 [03:06:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.52, 2.80, 3.36 [03:07:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 9.18, 9.10 [03:07:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 9.09, 8.79 [03:08:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.55, 3.74 [03:09:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.83, 9.16, 8.85 [03:11:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.21, 8.75, 8.95 [03:11:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.09, 10.08, 9.24 [03:12:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 4.48, 4.06 [03:12:51] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.55, 7.65, 8.39 [03:13:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.42, 9.17, 9.02 [03:14:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.67, 3.81 [03:15:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.00, 7.69, 8.49 [03:15:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.46, 9.18, 8.83 [03:16:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.93, 4.43, 4.06 [03:16:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 8.45, 8.57 [03:17:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.37, 9.17, 8.89 [03:19:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.09, 7.90, 8.47 [03:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.08, 3.31, 3.71 [03:20:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.36, 8.82, 8.66 [03:21:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.28, 8.95, 8.61 [03:22:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.41, 8.40, 8.53 [03:23:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.14, 8.91, 8.77 [03:23:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.81, 7.86, 8.41 [03:24:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.77, 2.41, 3.25 [03:24:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:24:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.32, 7.98, 8.36 [03:25:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 8.23, 8.38 [03:30:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.44, 8.98, 8.63 [03:31:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 9.53, 8.89 [03:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:33:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.95, 7.69, 8.34 [03:34:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.16, 8.16, 8.43 [03:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 8.78, 8.64 [03:39:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.14, 7.77, 8.35 [03:39:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 8.29, 8.47 [03:42:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.14, 9.86, 8.55 [03:44:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 9.42, 8.54 [03:45:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.89, 8.51, 8.55 [03:45:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 8.89, 8.66 [03:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.45, 9.25, 8.80 [03:48:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 8.22, 7.93 [03:48:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.26, 9.95, 8.95 [03:49:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.10, 8.80 [03:50:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.77, 8.04, 7.89 [03:50:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 9.22, 8.77 [03:50:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.82, 9.33, 8.86 [03:51:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.33, 9.73, 9.05 [03:52:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.89, 4.31, 3.49 [03:54:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.45, 9.27, 8.88 [03:54:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.45, 3.01 [03:56:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.70, 8.85, 8.79 [03:56:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.53, 10.31, 9.32 [03:56:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 3.23, 2.99 [03:57:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 9.64, 9.30 [03:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.83, 3.58 [04:00:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [04:00:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.23, 3.38 [04:00:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:01:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.78, 7.84, 8.42 [04:02:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.83, 7.69, 8.35 [04:03:10] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.22, 9.09, 9.04 [04:04:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.12, 8.99, 9.23 [04:05:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.90, 9.21, 9.08 [04:05:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:11:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.64, 9.64, 9.37 [04:13:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.14, 8.75, 9.09 [04:16:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 8.51, 8.26 [04:17:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 9.65, 9.33 [04:19:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 8.07, 8.45 [04:19:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 8.42, 8.22 [04:19:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.21, 8.57, 8.12 [04:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:21:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.24, 9.29, 8.43 [04:21:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.95, 9.85, 9.49 [04:22:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.99, 3.61 [04:22:06] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.07, 8.38, 8.34 [04:23:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.31, 9.99, 8.87 [04:23:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.35, 10.46, 9.76 [04:25:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.93, 8.45, 8.56 [04:25:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.14, 9.09, 8.67 [04:25:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.64, 8.59, 8.41 [04:25:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.26, 9.33, 9.43 [04:27:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.55, 7.93, 8.20 [04:29:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.81, 7.89, 8.35 [04:29:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.31, 8.26, 8.48 [04:33:31] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [04:33:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.68, 7.01, 8.34 [04:35:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.13, 3.02, 3.27 [04:37:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:42:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.36, 1.04 [04:53:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.40, 1.35, 1.08 [05:02:55] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [05:02:55] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406711101531982 seconds [05:14:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:19:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:24:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.93, 8.97, 7.55 [05:25:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.55, 1.78, 1.23 [05:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:27:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.49, 1.18 [05:30:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.01, 9.46, 8.52 [05:32:15] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [05:32:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.99, 8.20, 8.16 [05:32:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [05:32:25] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:35:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:01:19] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:21:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.47, 1.84, 1.22 [06:27:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.56, 1.87, 1.47 [06:31:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.34, 1.61, 1.45 [06:40:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.14, 8.16, 7.38 [06:41:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.93, 9.29, 7.65 [06:41:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:42:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.96, 7.39, 7.20 [06:43:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.86, 8.53, 7.57 [06:43:41] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [06:46:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:51:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.80, 9.11, 8.19 [06:51:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:51:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.03, 1.73, 1.52 [06:53:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 8.40, 8.04 [06:53:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.32, 1.66, 1.52 [07:01:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:10:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:15:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:15:41] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:29:43] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [07:29:44] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406681537628174 seconds [07:32:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [07:32:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:53:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 9.15, 7.79 [07:54:42] [02MirahezeMagic] 07Universal-Omega closed pull request 03#324: Fix usage of CentralAuthUser for MediaWiki 1.38.0+ - 13https://github.com/miraheze/MirahezeMagic/pull/324 [07:54:44] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/4553c0f0c85e...effaaa29cf7b [07:54:45] [02miraheze/MirahezeMagic] 07Universal-Omega 03effaaa2 - Fix usage of CentralAuthUser for MediaWiki 1.38.0+ (#324) [07:54:47] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeMagic [07:54:48] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [07:54:58] [02MirahezeMagic] 07RhinosF1 closed pull request 03#325: [py]: add CI - 13https://github.com/miraheze/MirahezeMagic/pull/325 [07:54:59] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03master [+2/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/effaaa29cf7b...76c1e10c1d43 [07:55:01] [02miraheze/MirahezeMagic] 07RhinosF1 0376c1e10 - [py]: add CI (#325) [07:55:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.68, 9.36, 8.01 [07:56:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.55, 8.25, 6.93 [07:57:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.31, 9.12, 8.12 [07:57:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.12, 9.42, 7.61 [07:58:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.48, 7.62, 6.85 [07:59:04] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:59:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.32, 8.08, 7.86 [07:59:47] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.63, 8.46, 7.48 [08:00:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:01:46] miraheze/MirahezeMagic - RhinosF1 the build passed. [08:05:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:07:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.22, 9.25, 7.93 [08:09:23] PROBLEM - bluearchive.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(h [08:09:24] 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405276536941528 seconds [08:11:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.61, 8.36, 7.86 [08:20:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.75, 8.75, 8.15 [08:22:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 8.28, 8.06 [08:30:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:33:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.06, 8.46, 7.87 [08:35:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.35, 7.81, 7.72 [08:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:39:11] RECOVERY - bluearchive.wiki - reverse DNS on sslhost is OK: SSL OK - bluearchive.wiki reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:39:39] [02CreateWiki] 07Universal-Omega closed pull request 03#301: Add phpunit tests for WikiManager - 13https://github.com/miraheze/CreateWiki/pull/301 [08:39:40] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/7a39921ce4ec...c9213be9f77c [08:39:42] [02miraheze/CreateWiki] 07Universal-Omega 03c9213be - Add phpunit tests for WikiManager (#301) [08:39:43] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [08:39:45] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [08:44:56] miraheze/CreateWiki - Universal-Omega the build passed. [09:06:18] [02CreateWiki] 07lens0021 commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1050667139 [09:06:41] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [09:10:03] miraheze/CreateWiki - lens0021 the build has errored. [09:12:07] [02CreateWiki] 07lens0021 commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1050671824 [09:17:34] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [09:20:34] miraheze/CreateWiki - lens0021 the build has errored. [09:24:47] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [09:30:13] miraheze/CreateWiki - lens0021 the build passed. [09:39:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.25, 3.44, 2.68 [09:40:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 2.94, 2.27 [09:42:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 2.84, 2.33 [09:43:43] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [09:44:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.77, 3.19 [09:46:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 3.36, 3.11 [09:50:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:56:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [09:56:34] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404694557189941 seconds [10:06:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.01, 8.65, 7.59 [10:07:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.48, 7.50, 7.30 [10:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:25:54] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [10:42:35] [discord] I’m getting a 404 wiki not found for Meta? [10:43:03] PROBLEM - mw102 MediaWiki Rendering on mw102 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.051 second response time [10:43:50] [discord] Now I’m not [10:44:06] [discord] I had it for about 10 minutes straight though [10:46:16] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:49:46] @Owen: please say if that happens again. You can also use the !_sre ping to get mediawiki team's quick attention [10:50:06] (Just type !_sre without the _) [10:51:42] Meta being a 404 is definitely an emergency [10:53:18] Reception123: depool mw102 please [10:53:52] RhinosF1: Mobile only until later this afternoon [10:54:19] paladox: you around? [10:55:12] Reception123: I can do it the unconventional way then [10:55:28] which is stop nginx and let cp* do it [10:55:55] RhinosF1: Is there no way to figure out what's wrong with it? If CosmicAlpha was around a reboot would've probably worked [10:56:41] Reception123: I have a guess [10:56:50] but I'm not sure why it would happen [10:57:05] What's the guess? [10:57:25] pm? [10:57:43] Sure [10:58:34] !log stop nginx & puppet on mw102 [10:58:59] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: connect to address 2a10:6740::6:108 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [10:59:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:00:49] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [11:01:05] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [11:01:10] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [11:01:11] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [11:01:19] PROBLEM - mw102 HTTPS on mw102 is CRITICAL: connect to address 2a10:6740::6:108 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [11:01:35] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [11:01:46] PROBLEM - mw102 Puppet on mw102 is WARNING: WARNING: Puppet is currently disabled, message: RhinosF1, last run 20 minutes ago with 0 failures [11:01:47] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw102 [11:02:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw102 [11:02:43] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw102 [11:05:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.14, 8.07, 7.22 [11:06:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 7.53, 6.51 [11:07:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:08:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.91, 7.11, 6.47 [11:09:29] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.landev.vn All nameservers failed to answer the query. [11:11:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.88, 7.96, 7.54 [11:12:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 8.16, 7.55 [11:12:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:14:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.32, 8.11, 7.61 [11:15:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.95, 8.83, 7.97 [11:16:49] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.324 second response time [11:16:54] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 0.320 second response time [11:17:05] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [11:17:10] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [11:17:11] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [11:17:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.77, 8.56, 7.62 [11:17:19] RECOVERY - mw102 HTTPS on mw102 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.015 second response time [11:17:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.82, 8.49, 7.95 [11:17:30] !log reset cache on mw102 and re-enable stuff [11:17:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:17:47] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:19:35] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [11:19:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:47] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [11:20:26] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [11:20:43] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 12 backends are healthy [11:21:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.06, 7.88, 7.61 [11:22:05] [02puppet] 07RhinosF1 opened pull request 03#2436: db: don't allow access to testglobal from non test servers - 13https://github.com/miraheze/puppet/pull/2436 [11:24:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:11] taavi: what are failed logins under? [11:32:14] Like what channel [11:32:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:34:37] RhinosF1: not anything by default in core iirc, wmf does it in a AuthManagerLoginAuthenticateAudit hook in our settings file [11:35:43] taavi: right [11:39:14] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.landev.vn could not be found [11:43:45] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Sat 02 Apr 2022 03:10:10 GMT +0000. [11:52:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.37, 8.52, 7.63 [11:53:58] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:53:58] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406702280044556 seconds [11:54:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.20, 8.38, 7.68 [11:55:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:56:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 8.70, 7.68 [11:58:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.45, 7.91, 7.52 [11:59:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.21, 8.48, 7.74 [12:01:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.92, 7.93, 7.63 [12:01:19] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:02:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.03, 8.88, 8.04 [12:04:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.61, 7.69, 7.70 [12:04:54] [02puppet] 07RhinosF1 opened pull request 03#2437: Update mediawiki.conf - 13https://github.com/miraheze/puppet/pull/2437 [12:05:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 8.86, 8.08 [12:06:43] [02puppet] 07paladox closed pull request 03#2437: Update mediawiki.conf - 13https://github.com/miraheze/puppet/pull/2437 [12:06:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c8842145294...3f018af8b218 [12:06:46] [02miraheze/puppet] 07RhinosF1 033f018af - Update mediawiki.conf (#2437) [12:09:00] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.landev.vn All nameservers failed to answer the query. [12:10:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:11:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.78, 9.80, 8.37 [12:13:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.17, 8.65, 8.02 [12:15:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.60, 7.79, 7.78 [12:15:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.42, 7.74, 8.04 [12:15:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 9.18, 8.43 [12:19:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 8.80, 8.40 [12:22:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.69, 8.25, 7.62 [12:23:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.35, 8.33, 8.31 [12:24:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 8.06, 7.63 [12:25:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:25:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.68, 1.73, 1.35 [12:25:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.37, 7.94, 8.26 [12:27:40] PROBLEM - www.connorjwat.net - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/li [12:27:40] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query connorjwat.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:27:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.49, 1.66, 1.37 [12:27:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 2.04, 1.45 [12:29:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.62, 1.85, 1.45 [12:30:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:31:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.62, 1.42 [12:32:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:35:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.23, 1.91, 1.57 [12:38:44] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.landev.vn could not be found [12:41:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.86, 1.67 [12:43:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.24, 2.61 [12:45:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 2.95, 2.58 [12:45:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.97, 1.47, 1.56 [12:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:53:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.97, 1.73, 1.54 [12:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:57:30] PROBLEM - www.connorjwat.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.connorjwat.net All nameservers failed to answer the query. [12:57:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.33, 1.58, 1.53 [13:01:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [13:01:19] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:08:30] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.landev.vn All nameservers failed to answer the query. [13:12:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a51991fa587c [13:12:21] [02miraheze/mw-config] 07RhinosF1 03a51991f - Logging: adjust 2 channels [13:12:22] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [13:12:29] [02mw-config] 07RhinosF1 opened pull request 03#4483: Logging: adjust 2 channels - 13https://github.com/miraheze/mw-config/pull/4483 [13:12:58] paladox, Reception123: is ^ ok [13:13:44] miraheze/mw-config - RhinosF1 the build passed. [13:15:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 2.99, 2.62 [13:16:09] [02mw-config] 07RhinosF1 closed pull request 03#4483: Logging: adjust 2 channels - 13https://github.com/miraheze/mw-config/pull/4483 [13:16:11] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/93d759472bf0...01c6146f0c0e [13:16:12] [02miraheze/mw-config] 07RhinosF1 0301c6146 - Logging: adjust 2 channels (#4483) [13:16:14] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [13:16:15] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [13:17:19] miraheze/mw-config - RhinosF1 the build passed. [13:17:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.56, 2.51 [13:17:42] !log [@test101] starting deploy of {'config': True} to all [13:17:43] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [13:17:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:52] !log [@mwtask111] starting deploy of {'config': True} to all [13:22:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:01] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [13:23:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:22] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [13:25:57] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.59, 1.89, 1.35 [13:27:21] RECOVERY - www.connorjwat.net - reverse DNS on sslhost is OK: SSL OK - www.connorjwat.net reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:27:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.09, 1.53, 1.28 [13:38:16] PROBLEM - wiki.landev.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.landev.vn could not be found [14:06:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.19, 2.20, 1.32 [14:06:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.33, 2.84 [14:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:08:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.67, 1.23 [14:08:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.14, 2.83 [14:09:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.48, 3.03 [14:14:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.00, 3.30 [14:16:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.79, 3.33 [14:17:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.43, 8.16, 7.26 [14:18:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.81, 3.39 [14:21:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.00, 8.11, 7.52 [14:22:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.40, 3.31 [14:23:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.01, 8.62, 8.02 [14:27:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.72, 1.07 [14:28:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.80, 3.81 [14:28:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.71, 4.26, 3.69 [14:29:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.32, 2.84 [14:29:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.46, 1.04 [14:29:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.50, 8.07, 8.01 [14:30:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:30:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.79, 4.26, 4.01 [14:31:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.33, 3.31, 2.89 [14:32:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.89, 3.67 [14:32:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.59, 3.80 [14:36:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.50, 2.74, 3.26 [14:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:38:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.58, 2.43, 3.22 [14:39:38] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.71, 1.69, 1.26 [14:40:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 8.34, 8.04 [14:41:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.17, 1.59, 1.29 [14:42:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.83, 3.63 [14:44:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.52, 3.53 [14:44:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.34, 7.90, 7.93 [14:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:50:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.74, 3.57 [14:50:43] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.76, 8.60, 7.74 [14:53:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [14:53:24] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406677961349487 seconds [14:53:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.81, 8.70, 7.93 [14:54:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.92, 7.78, 7.64 [14:55:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 8.04, 7.78 [14:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:04:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.89, 3.93 [15:16:57] PROBLEM - wiki.landev.vn - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.landev.vn and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:23:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [15:24:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.80, 3.24 [15:25:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:29:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.20, 7.80, 6.94 [15:31:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.53, 7.34, 6.88 [15:32:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 4.03, 3.58 [15:32:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.99, 3.29, 2.85 [15:39:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.97, 8.49, 7.52 [15:40:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 2.87, 2.75 [15:43:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.46, 7.70, 7.51 [15:48:08] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.30, 1.98, 1.44 [15:48:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.57, 3.93 [15:50:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.75, 1.54, 1.34 [15:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:53:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [15:53:24] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40636420249939 seconds [15:55:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.31, 1.70, 1.36 [15:55:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.62, 8.81, 7.93 [15:57:20] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.43, 1.50, 1.32 [15:57:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.00, 8.37, 7.87 [16:00:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 2.85, 3.38 [16:06:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.56, 3.53 [16:08:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:10:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.74, 3.57 [16:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:16:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.70, 3.64 [16:21:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:22:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 2.98, 3.31 [16:26:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 1.75, 1.20 [16:26:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:28:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.33, 1.11 [16:29:35] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.52, 3.10 [16:30:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.77, 3.40 [16:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:33:28] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.42, 3.14 [16:37:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.79, 3.38, 3.21 [16:38:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.97, 3.74 [16:41:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:44:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 8.70, 7.98 [16:44:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.72, 3.67 [16:46:12] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1051016908 [16:46:26] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1051017093 [16:46:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.42, 3.57 [16:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:47:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 8.19, 7.54 [16:49:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.74, 8.19, 7.63 [16:50:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.18, 4.00, 3.72 [16:51:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.01, 7.99, 7.62 [16:51:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.67, 8.42, 7.81 [16:51:51] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.22, 7.91, 8.00 [16:52:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.54, 3.58 [16:53:22] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [16:55:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.02, 9.05, 7.95 [16:56:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.57, 8.26, 7.47 [16:56:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.06, 3.37 [16:57:39] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.42, 9.37, 8.60 [16:57:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.27, 8.86, 8.03 [16:59:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 9.15, 8.30 [16:59:34] [02CreateWiki] 07Universal-Omega reviewed pull request 03#283 commit - 13https://github.com/miraheze/CreateWiki/pull/283#discussion_r814936980 [16:59:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.31, 8.92, 8.51 [17:01:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.39, 8.70, 8.23 [17:01:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.45, 8.08, 7.95 [17:02:07] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 8.28, 7.82 [17:02:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.94, 3.63 [17:02:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.49, 3.17 [17:03:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.97, 4.17, 3.26 [17:03:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.50, 8.61 [17:03:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.72, 8.45, 8.44 [17:04:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.26, 3.12 [17:05:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.85, 3.26 [17:06:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.58, 3.58 [17:07:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.30, 3.27, 3.12 [17:07:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.99, 8.37, 8.23 [17:07:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:10:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 8.16, 7.88 [17:10:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.09, 3.37 [17:10:49] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 9.45, 8.59 [17:12:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.80, 7.80, 7.79 [17:12:33] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [17:12:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.01, 9.66, 8.80 [17:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:15:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.47, 8.76 [17:20:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.07, 2.89 [17:21:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 9.80, 9.18 [17:22:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.70, 2.78 [17:22:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.92, 8.08, 8.41 [17:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:24:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.24, 3.56, 3.37 [17:27:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.63, 9.35, 9.66 [17:29:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:30:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.53, 3.15 [17:30:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 8.45, 8.14 [17:32:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 3.97, 3.35 [17:32:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.59, 3.12 [17:32:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.16, 8.23, 8.11 [17:32:49] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1051052711 [17:33:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 10.71, 10.12 [17:34:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.57, 3.17 [17:35:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 9.77, 9.85 [17:35:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.04, 9.57, 9.16 [17:36:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.39, 3.16 [17:36:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.07, 8.61, 8.32 [17:36:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.44, 9.38, 8.63 [17:37:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.60, 9.22 [17:38:18] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Convert to PSR-4 and use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1051057105 [17:40:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.60, 3.28 [17:40:32] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:40:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.38, 9.47, 8.87 [17:42:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.55, 3.35, 3.24 [17:42:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.77, 3.59 [17:42:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.41, 8.48, 8.40 [17:44:39] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.74, 9.70, 9.06 [17:46:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.93, 9.35, 9.00 [17:50:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.15, 2.47, 3.14 [17:50:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 7.82, 8.47 [17:50:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.42, 3.86 [17:50:36] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4179 bytes in 0.057 second response time [17:51:33] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [17:51:38] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 14.08, 22.43, 11.84 [17:51:39] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [17:51:51] F [17:51:53] !sre [17:51:57] Reception123, paladox [17:52:46] JohnLewis: ^ [17:53:08] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 6.89, 4.51, 2.51 [17:53:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 4.80, 6.41, 8.07 [17:53:19] !log restart mariadb on db121 [17:53:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [17:53:24] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407044172286987 seconds [17:53:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.55, 7.41, 8.48 [17:53:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:55:02] I'm guessing you guys are already aware of some of the technical issues? [17:55:08] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 5.20, 4.64, 2.80 [17:55:30] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 137 Threads: 50 Questions: 8969 Slow queries: 472 Opens: 1136 Open tables: 1130 Queries per second avg: 65.467 [17:55:36] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.104 second response time [17:56:26] [discord] Oh, it's back. [17:56:27] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.179 second response time [17:56:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.65, 3.76 [17:57:07] darkmatterman450: we're aware [17:57:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.70, 4.32, 2.91 [17:57:11] paladox: why it crash [17:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.59, 3.73 [18:00:11] JohnLewis: it seems that https://github.com/miraheze/puppet/blob/0dc92cd997f933d4d70ea1a4df1c6d7bab9d67d8/modules/role/manifests/prometheus.pp#L147 isn't collecting the db hosts [18:00:11] [url] puppet/prometheus.pp at 0dc92cd997f933d4d70ea1a4df1c6d7bab9d67d8 · miraheze/puppet · GitHub | github.com [18:00:28] the file is empty of hostnames and the dashboard for mariadb doesn't have any hosts either [18:00:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.51, 4.03, 3.85 [18:00:59] Check if Prometheus::Exporter::Mariadb is correct [18:01:35] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.92, 4.88, 7.29 [18:02:20] guess it's because it uses a define JohnLewis [18:02:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.66, 3.39 [18:02:27] and https://github.com/miraheze/puppet/blob/b71a1d19265dde29d07a1004247bfcdaf69680a7/modules/prometheus/manifests/class.pp#L6 looks for class [18:02:28] [url] puppet/class.pp at b71a1d19265dde29d07a1004247bfcdaf69680a7 · miraheze/puppet · GitHub | github.com [18:02:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.24, 2.98 [18:02:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:02:42] That would explain it [18:03:34] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.57, 4.31, 6.79 [18:04:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.32, 3.21, 3.00 [18:06:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.79, 3.90 [18:07:36] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:08:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.28, 3.32 [18:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:14:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 2.76, 3.33 [18:23:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [18:24:30] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.04, 2.17, 1.49 [18:26:28] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.65, 1.93, 1.49 [18:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:30:23] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.16, 2.01, 1.61 [18:36:16] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.79, 1.89, 1.70 [18:41:06] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.21, 8.69, 7.76 [18:41:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.52, 3.14 [18:42:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.38, 1.62, 1.63 [18:43:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 8.48, 7.78 [18:43:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.22, 3.07 [18:45:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.47, 7.69, 7.57 [18:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:46:27] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4179 bytes in 0.097 second response time [18:47:20] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.90, 7.93, 5.69 [18:47:30] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [18:47:35] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [18:47:36] !_sre ^ [18:47:36] only to be used when you need quick attention from as much of sre as possible [18:49:20] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.73, 5.48, 5.05 [18:51:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.54, 3.27 [18:53:22] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.00, 3.18, 3.19 [18:53:53] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.90, 1.78, 1.65 [18:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:55:51] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.40, 1.64, 1.62 [18:55:59] JohnLewis: ^ [18:56:13] I don’t have access right now, [18:56:47] Agent: remove the _ [18:57:34] I don’t monitor !sre [18:58:04] paladox: mediawiki team do and please don't use it outside of emergencies [18:59:46] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.91, 1.75, 1.66 [19:03:46] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.47, 1.63, 1.64 [19:07:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/b16f16d99b44 [19:07:07] [02miraheze/CreateWiki] 07Universal-Omega 03b16f16d - Add more wiki deletion tests [19:07:08] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [19:07:10] [02CreateWiki] 07Universal-Omega opened pull request 03#303: Add more wiki deletion tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:07:28] paladox, what's up? [19:07:51] Voidwalker: db121 MySQL needs restarting [19:08:24] !log db121 restart mysql [19:08:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:35] Thanks! [19:08:43] [02miraheze/CreateWiki] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b16f16d99b44...57c54aa747d2 [19:08:45] [02miraheze/CreateWiki] 07github-actions 0357c54aa - CI: lint code to MediaWiki standards [19:08:46] [02CreateWiki] 07github-actions[bot] synchronize pull request 03#303: Add more wiki deletion tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:09:30] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 75 Threads: 90 Questions: 5756 Slow queries: 356 Opens: 1105 Open tables: 1099 Queries per second avg: 76.746 [19:09:35] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.266 second response time [19:09:51] miraheze/CreateWiki - Universal-Omega the build has errored. [19:10:27] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.255 second response time [19:12:27] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/57c54aa747d2...1ec44975bd4b [19:12:29] [02miraheze/CreateWiki] 07Universal-Omega 031ec4497 - Update WikiManagerTest.php [19:12:30] [02CreateWiki] 07Universal-Omega synchronize pull request 03#303: Add more wiki deletion tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:13:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.36, 1.91, 1.26 [19:15:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.61, 1.45, 1.17 [19:16:05] miraheze/CreateWiki - Universal-Omega the build has errored. [19:27:26] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1ec44975bd4b...d60da2f11078 [19:27:27] [02miraheze/CreateWiki] 07Universal-Omega 03d60da2f - Update WikiManagerTest.php [19:27:29] [02CreateWiki] 07Universal-Omega synchronize pull request 03#303: Add more wiki deletion tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:31:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:32:44] miraheze/CreateWiki - Universal-Omega the build passed. [19:34:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.18, 9.04, 7.91 [19:36:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:36:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.35, 8.28, 7.75 [19:40:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.32, 8.66, 8.05 [19:42:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.73, 8.20, 7.96 [19:47:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.02, 1.85, 1.56 [19:47:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:49:19] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.33, 1.36, 1.41 [19:49:33] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d60da2f11078...11ec7a21c806 [19:49:34] [02miraheze/CreateWiki] 07Universal-Omega 0311ec7a2 - Add testDeleteRecreate and testCreateExists [19:49:36] [02CreateWiki] 07Universal-Omega synchronize pull request 03#303: Add more wiki deletion tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:49:47] [02CreateWiki] 07Universal-Omega edited pull request 03#303: Add more WikiManager phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:52:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:53:04] miraheze/CreateWiki - Universal-Omega the build has errored. [19:53:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [19:53:24] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404675722122192 seconds [19:54:45] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/11ec7a21c806...59a8deb61352 [19:54:46] [02miraheze/CreateWiki] 07Universal-Omega 0359a8deb - Update WikiManagerTest.php [19:54:48] [02CreateWiki] 07Universal-Omega synchronize pull request 03#303: Add more WikiManager phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/303 [19:57:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:59:53] miraheze/CreateWiki - Universal-Omega the build passed. [20:00:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.35, 3.00 [20:02:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:02:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.00, 3.20, 2.99 [20:05:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5515c188f92b [20:05:14] [02miraheze/puppet] 07paladox 035515c18 - prometheus::class support define [20:05:15] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [20:05:17] [02puppet] 07paladox opened pull request 03#2438: prometheus::class support define - 13https://github.com/miraheze/puppet/pull/2438 [20:05:37] [02puppet] 07paladox edited pull request 03#2438: prometheus::class: support define - 13https://github.com/miraheze/puppet/pull/2438 [20:05:42] [02puppet] 07paladox closed pull request 03#2438: prometheus::class: support define - 13https://github.com/miraheze/puppet/pull/2438 [20:05:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3f018af8b218...9fd7f88d3c98 [20:05:45] [02miraheze/puppet] 07paladox 039fd7f88 - prometheus::class: support define (#2438) [20:05:47] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [20:05:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [20:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:08:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.46, 3.16 [20:10:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.28, 3.38, 3.17 [20:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:14:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.31, 3.59 [20:16:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.88, 3.52 [20:18:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.53, 8.79, 8.11 [20:20:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.40, 8.05, 7.92 [20:20:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.90, 3.56 [20:22:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.93, 3.63 [20:27:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.19, 8.40, 8.09 [20:28:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.83, 8.54, 8.20 [20:29:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:30:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 8.65, 7.70 [20:30:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 2.99, 3.34 [20:30:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.67, 8.87, 7.93 [20:32:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.24, 7.65, 7.60 [20:33:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.43, 8.41, 8.29 [20:34:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.66, 8.36, 7.79 [20:34:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.34, 8.93, 8.33 [20:36:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.47, 8.21, 8.24 [20:40:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.39, 8.46, 8.36 [20:41:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.11, 8.87, 8.00 [20:43:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.84, 8.39, 7.92 [20:43:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.20, 8.52, 8.37 [20:44:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.23, 8.88, 8.58 [20:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:45:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.43, 7.79, 8.13 [20:46:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.16, 8.39, 8.45 [20:52:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.44, 3.21 [20:53:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:54:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.20, 3.16 [20:57:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.66, 3.39 [20:58:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:59:42] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/59a8deb61352...8fbcf886a29b [20:59:43] [02miraheze/CreateWiki] 07Universal-Omega 038fbcf88 - Add checkDatabaseName tests [20:59:45] [02CreateWiki] 07Universal-Omega synchronize pull request 03#303: Add more WikiManager phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/303 [21:00:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.66, 8.21, 7.99 [21:00:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 8.60, 8.35 [21:02:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.38, 7.04, 7.61 [21:02:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.62, 8.07, 8.17 [21:04:26] miraheze/CreateWiki - Universal-Omega the build passed. [21:05:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.76, 4.18, 3.70 [21:06:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:07:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.90, 3.66 [21:09:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 4.63, 3.95 [21:10:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.47, 2.95 [21:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:12:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 2.82, 2.76 [21:12:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.19, 8.53, 7.93 [21:13:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.78, 3.76 [21:14:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.74, 8.16, 7.86 [21:17:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.30, 4.14, 3.84 [21:18:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:21:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.02, 3.98, 3.89 [21:23:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [21:23:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:27:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.38, 3.58 [21:27:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.35, 2.89 [21:27:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:29:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.39, 3.35, 2.95 [21:30:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.64, 3.01 [21:32:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 3.30, 2.96 [21:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:33:14] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [21:33:14] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:34:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8fbcf886a29b...76ba7f59c4a6 [21:34:07] [02miraheze/CreateWiki] 07Universal-Omega 0376ba7f5 - Update WikiManagerTest.php [21:34:08] [02CreateWiki] 07Universal-Omega synchronize pull request 03#303: Add more WikiManager phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/303 [21:36:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.18, 9.47, 7.91 [21:38:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.61, 9.05, 7.95 [21:39:14] miraheze/CreateWiki - Universal-Omega the build passed. [21:39:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:40:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.45, 3.20 [21:40:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.07, 8.27, 7.79 [21:42:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.33, 3.19 [21:42:33] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.68, 9.11, 8.29 [21:44:29] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.04, 8.50, 8.18 [21:44:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:45:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:10] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:47] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:02] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:03] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:03] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:11] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:13] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 9.688 second response time [21:47:13] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:31] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:35] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:14] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:36] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.91, 2.97, 3.76 [21:49:14] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.786 second response time [21:49:19] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time [21:49:30] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.418 second response time [21:49:34] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.302 second response time [21:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:49:52] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 0.353 second response time [21:49:54] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [21:50:03] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.381 second response time [21:50:10] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.543 second response time [21:50:12] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.593 second response time [21:50:17] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [21:50:32] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22297 bytes in 0.374 second response time [21:52:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 2.53, 3.40 [21:53:24] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [21:53:24] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404273509979248 seconds [21:55:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.94, 9.08, 8.17 [21:56:05] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.68, 8.69, 8.28 [21:58:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.86, 3.69, 3.20 [21:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.37, 3.52 [21:59:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.83, 9.21, 8.47 [21:59:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.72, 7.92, 8.07 [22:00:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.03, 3.37 [22:01:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.03, 9.97, 8.84 [22:02:09] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [22:02:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.49, 3.21 [22:03:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.13, 9.58, 8.83 [22:04:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 4.09, 3.47 [22:05:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 8.72, 7.99 [22:06:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.78, 3.44 [22:07:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.74, 8.39, 8.50 [22:07:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.81, 7.78, 7.73 [22:10:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.33, 3.37 [22:11:08] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.37, 3.23, 1.78 [22:12:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.14, 8.85, 8.48 [22:15:08] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.64, 1.81, 1.52 [22:17:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.83, 1.52, 1.45 [22:22:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.33, 8.47, 8.47 [22:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:32:59] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [22:35:36] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9ab5ef745a61 [22:35:38] [02miraheze/mw-config] 07RhinosF1 039ab5ef7 - POC: stop bots accessing some special pages [22:35:39] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://github.com/miraheze/mw-config [22:44:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:48:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.22, 2.98 [22:49:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:50:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.09, 2.97 [22:51:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.09, 8.63, 8.04 [22:53:10] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.13, 8.11, 7.93 [22:58:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.30, 3.06 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:33] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 2.94, 2.95 [23:07:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.37, 2.92 [23:07:37] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 455s [23:07:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:09:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.76, 3.69, 3.09 [23:15:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.12, 3.08 [23:19:52] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1189s [23:19:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:20] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [23:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:26:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.89, 7.86, 7.08 [23:28:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:28:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.83, 3.35 [23:28:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.89, 7.12, 6.90 [23:30:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 4.09, 3.50 [23:30:48] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [23:32:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.84, 3.48 [23:34:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 15.08, 5.72, 2.62 [23:38:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.16, 3.29 [23:42:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.64, 1.74, 1.86 [23:44:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.45, 1.30, 1.68 [23:48:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 11.50, 4.87, 3.09 [23:49:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:51:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.44, 8.30, 7.24 [23:53:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 8.33, 7.37 [23:53:54] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(ho [23:53:54] 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406949281692505 seconds [23:55:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.49, 8.04, 7.38 [23:59:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki