[00:00:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 8.76, 8.48 [00:00:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.16, 7.91, 8.08 [00:00:41] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:00:41] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:00:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.76, 3.70 [00:01:12] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.79, 3.73 [00:02:52] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 2150 MB (22% inode=80%); [00:03:01] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:03:13] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:03:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 691 failures. Last run 2 minutes ago with 691 failures. Failed resources (up to 3 shown): File[en.gyaanipedia.com_private],File[hi.gyaanipedia.com],File[hi.gyaanipedia.com_private],File[pa.gyaanipedia.com] [00:03:38] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:04:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.84, 7.69, 8.18 [00:05:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.63, 3.46 [00:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:06:36] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.88, 5.05, 4.00 [00:08:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 4.01, 3.74 [00:09:03] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.71, 8.76, 6.31 [00:09:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.66, 3.25 [00:10:17] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [00:10:38] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.031 second response time [00:10:40] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [00:10:55] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [00:10:57] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 3.819 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [00:11:05] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:11:11] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [00:11:33] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 24296 MB (36% inode=89%); [00:11:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 3.97, 3.42 [00:11:41] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [00:11:41] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:12:07] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.005189746618 secs [00:12:13] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [00:12:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.65, 3.67 [00:13:30] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.563 second response time [00:13:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.81, 3.42 [00:13:54] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.835 second response time [00:14:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.16, 8.93, 8.62 [00:14:06] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 8.717 second response time [00:14:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 4.05, 3.81 [00:16:04] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:17:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.48, 3.12, 3.24 [00:17:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.97, 3.97 [00:17:45] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:18:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 8.05, 8.35 [00:18:04] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:18:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.76, 3.74 [00:19:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 4.23, 4.07 [00:19:45] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:20:00] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:20:21] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:21] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.92, 3.80 [00:21:03] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.67, 5.16, 5.95 [00:21:11] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:21:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.85, 3.96 [00:21:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 8.54, 8.48 [00:22:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.79, 3.76 [00:22:48] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:03] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.07, 5.51, 5.97 [00:23:14] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:38] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:23:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.36, 9.25, 8.75 [00:24:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.98, 3.82 [00:24:46] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.31, 9.09, 8.28 [00:25:03] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.15, 5.07, 5.76 [00:25:18] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:25:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.87, 3.92 [00:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:26:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.69, 3.75 [00:26:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.22, 8.44, 8.13 [00:27:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.32, 3.20 [00:27:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.18, 8.89, 8.72 [00:29:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.68, 3.36 [00:31:03] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.29, 7.01, 6.22 [00:31:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.64, 3.85 [00:31:22] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:31:55] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [00:31:58] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [00:31:58] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset -0.01059105992 secs [00:32:11] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [00:32:26] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 7.926 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [00:33:03] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.74, 5.79, 5.88 [00:33:12] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 24297 MB (36% inode=89%); [00:33:16] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [00:33:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.95, 3.94 [00:33:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.85, 4.20, 3.57 [00:33:34] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [00:33:51] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.84, 7.93, 8.37 [00:33:52] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:34:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.72, 1.38 [00:35:03] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 18.90, 8.95, 6.91 [00:36:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.52, 1.35 [00:37:32] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:37:40] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.87, 9.17, 8.84 [00:38:47] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:39:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.56, 3.81 [00:39:11] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:39:12] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:39:34] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:39:41] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [00:39:46] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.49, 8.98 [00:41:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 4.32, 4.08 [00:41:15] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:41:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.78, 3.74 [00:41:38] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:41:44] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:41:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 8.70, 8.09 [00:42:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.47, 9.01, 8.42 [00:42:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.67, 2.83, 3.27 [00:43:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.41, 3.55, 3.82 [00:43:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 9.85, 9.28 [00:43:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.11, 8.49, 8.09 [00:44:02] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.22, 8.27, 8.22 [00:44:59] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:45:40] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:47:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.32, 3.09, 3.40 [00:47:35] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [00:47:38] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset -0.0009322166443 secs [00:47:40] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 3.094 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [00:47:40] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 24296 MB (36% inode=89%); [00:48:07] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [00:48:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.96, 8.64, 8.37 [00:50:22] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:50:23] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [00:50:25] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [00:50:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 3.38, 3.54 [00:51:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.47, 9.33, 9.05 [00:52:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.59, 7.78, 8.11 [00:52:35] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:52:42] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [00:52:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.19, 3.46 [00:53:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 8.85, 8.90 [00:54:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.05, 3.03, 3.36 [00:55:02] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.831 second response time [00:55:13] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22345 bytes in 5.695 second response time [00:55:41] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.528 second response time [00:56:10] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.011 second response time [00:56:18] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:56:37] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.045 second response time [00:56:40] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.027 second response time [00:57:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 8.44, 7.92 [00:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.91, 9.47, 9.12 [00:59:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.98, 8.08, 7.84 [00:59:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.50, 3.95, 3.37 [00:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.62, 8.84, 8.93 [01:00:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.42, 3.28 [01:01:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.75, 3.38 [01:02:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 2.94, 3.11 [01:03:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.85, 3.45 [01:03:57] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:05:01] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:02] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.83, 3.49 [01:05:19] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:20] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.66, 9.23 [01:05:37] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.86, 3.55 [01:06:13] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:20] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:06:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.45, 3.30 [01:06:56] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.262 second response time [01:07:00] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.025 second response time [01:07:19] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 3.285 second response time [01:07:22] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [01:07:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.78, 3.56 [01:07:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.53, 8.06, 7.60 [01:08:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.08, 3.18 [01:09:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.11, 3.30 [01:09:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.38, 8.84, 9.00 [01:09:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.00, 7.89, 7.58 [01:10:50] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:15] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:25] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:43] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:44] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:34] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 1.150 second response time [01:12:44] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:12:44] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:12:46] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.131 second response time [01:12:49] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:13:11] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.679 second response time [01:13:24] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.379 second response time [01:13:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.15, 3.28, 3.40 [01:13:42] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 1.695 second response time [01:13:43] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [01:13:51] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:14:34] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:44] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [01:16:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.57, 3.32 [01:17:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.20, 4.10, 3.70 [01:17:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.25, 7.77, 8.44 [01:18:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.12, 3.18 [01:19:03] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:06] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:26] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.28, 3.35, 3.47 [01:19:48] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:03] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:05] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:02] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.094 second response time [01:21:04] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.988 second response time [01:21:23] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 2.291 second response time [01:21:47] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 1.364 second response time [01:22:00] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [01:22:02] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.900 second response time [01:23:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.58, 3.51 [01:27:57] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:29:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.33, 3.44, 3.52 [01:29:17] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:30:11] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:24] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:33] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:30:48] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:56] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:07] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:32:49] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:32:53] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:34:02] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:34:04] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:35:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.38, 3.40 [01:35:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.71, 8.86, 8.42 [01:35:33] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:35:40] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:36:19] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:39:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.16, 2.97, 3.24 [01:41:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.36, 8.40, 8.42 [01:41:59] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:42:18] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [01:42:18] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [01:42:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:42:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.80, 8.70, 7.69 [01:43:32] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [01:43:34] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [01:44:04] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 24264 MB (36% inode=89%); [01:44:05] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [01:44:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.88, 9.18, 8.36 [01:44:32] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset -0.01116222143 secs [01:44:50] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 8.089 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [01:44:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.97, 7.94, 7.53 [01:45:11] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [01:45:13] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 4.282 second response time [01:45:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.03, 9.60, 8.89 [01:46:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.51, 8.49, 8.21 [01:46:38] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:46:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.92, 4.66, 3.89 [01:46:58] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 51 minutes ago with 0 failures [01:47:05] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.335 second response time [01:47:08] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.031 second response time [01:47:55] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22345 bytes in 3.312 second response time [01:48:04] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.328 second response time [01:48:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.67, 3.62 [01:49:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 9.43, 9.01 [01:50:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.60, 8.73 [01:50:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.82, 3.66 [01:52:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.44, 3.36, 3.52 [01:53:03] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.83, 2.55, 5.44 [01:55:03] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.16, 2.18, 4.95 [01:56:13] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [01:57:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.04, 7.71, 8.39 [02:00:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 2.92, 3.27 [02:02:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.42, 3.12 [02:02:32] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:04:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.65, 3.23 [02:05:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.17, 10.07, 8.38 [02:05:58] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.04, 9.41, 8.63 [02:07:52] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.021 second response time [02:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:08:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.58, 3.33 [02:09:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.49, 8.64, 8.42 [02:10:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.59, 3.22, 3.22 [02:10:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.18, 9.17, 8.64 [02:11:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.32, 9.04, 8.21 [02:12:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.47, 8.30, 8.39 [02:13:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 9.81, 9.05 [02:13:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.29, 8.34, 8.06 [02:14:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.73, 3.44 [02:15:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.78, 1.38 [02:15:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.56, 3.41 [02:17:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.58, 1.35 [02:17:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.38, 3.17 [02:18:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:19:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.77, 2.68, 2.93 [02:19:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.14, 8.80 [02:19:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 2.96, 3.20 [02:20:17] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:28] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:35] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:52] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.96, 8.14, 8.50 [02:21:44] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 9.06, 8.82 [02:21:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.64, 9.07, 8.63 [02:22:16] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:43] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:22:55] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:22:57] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:23:04] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:23:05] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:23:43] PROBLEM - mwtask111 HTTPS on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:45] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:24:16] PROBLEM - mwtask111 NTP time on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:24:45] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:25:19] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:25:25] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:25:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.32, 8.50, 8.50 [02:25:58] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [02:25:58] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404378652572632 seconds [02:28:31] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 32 minutes ago with 0 failures [02:28:35] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [02:28:36] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [02:28:43] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [02:28:49] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22345 bytes in 1.561 second response time [02:28:59] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.386 second response time [02:29:20] RECOVERY - mwtask111 NTP time on mwtask111 is OK: NTP OK: Offset 0.006228208542 secs [02:29:21] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.319 second response time [02:29:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.58, 9.48, 8.90 [02:29:50] RECOVERY - mwtask111 HTTPS on mwtask111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 553 bytes in 0.021 second response time [02:29:51] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.024 second response time [02:30:08] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:30:14] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [02:30:32] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.332 second response time [02:30:48] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.514 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [02:30:50] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 24376 MB (36% inode=89%); [02:30:55] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [02:31:39] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:32:39] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(ho [02:32:39] 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405963182449341 seconds [02:33:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.10, 7.46, 8.23 [02:33:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 9.60, 9.13 [02:34:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.92, 8.68, 8.37 [02:34:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.53, 3.35 [02:36:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 10.16, 8.96 [02:36:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 3.84, 3.46 [02:37:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.32, 9.58, 8.88 [02:38:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 9.39, 8.81 [02:39:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.45, 8.62, 8.60 [02:40:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.80, 8.70, 8.31 [02:41:23] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.57, 8.03, 8.40 [02:42:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.81, 8.49, 8.29 [02:43:20] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [02:43:21] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:43:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.75, 7.67, 8.42 [02:44:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.26, 3.19, 3.41 [02:48:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:50:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.78, 7.88, 8.33 [02:50:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 2.93, 3.31 [02:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.80, 8.60, 8.47 [02:53:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.19, 8.39, 8.41 [02:55:43] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.29, 3.78, 3.49 [03:01:35] SRE, getting persistent 503s on `loginwiki` [03:02:08] ^ Seems to be transitory and resolved, for now [03:02:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.52, 1.28 [03:02:39] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:02:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.41, 3.39 [03:03:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.56, 8.96, 8.56 [03:04:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.13, 1.36, 1.24 [03:04:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.07, 8.59, 8.23 [03:04:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.23, 3.32 [03:05:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.78, 8.54 [03:06:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.34, 8.33, 8.17 [03:07:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.92, 8.17, 8.35 [03:12:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.00, 9.38, 8.66 [03:12:55] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [03:13:14] [discord] ``https://unpkg.com/*`` needs to be whitelisted for cross-origin domain as the Evelution Skin uses data from that site for the GPU detection tool [03:13:32] [discord] And as such, transparency effects never show on Evelution in Miraheze [03:13:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 4.01, 3.65 [03:15:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.49, 3.52 [03:18:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.39, 0.52, 1.86 [03:19:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.12, 4.16, 3.76 [03:19:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.16, 8.37, 8.47 [03:20:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.18, 0.40, 1.65 [03:21:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.84, 3.69 [03:22:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 1.80, 1.40 [03:24:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.99, 8.99, 8.55 [03:24:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.42, 1.31 [03:26:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:09] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.18, 8.35, 8.37 [03:27:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.66, 3.60 [03:29:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.31, 3.47 [03:31:36] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.71, 2.71, 3.23 [03:33:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:34:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.81, 8.25, 8.10 [03:34:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.38, 1.67, 1.53 [03:40:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 8.04, 8.07 [03:40:48]