[00:04:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.64, 7.57, 6.53 [00:06:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.55, 7.38, 6.57 [00:10:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 5.35, 6.43, 6.38 [01:09:06] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.36, 6.91, 6.48 [01:13:01] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 7.47, 6.80 [01:14:59] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.16, 6.99, 6.70 [01:16:56] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.47, 7.41, 6.89 [01:22:49] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.66, 7.67, 7.35 [01:34:34] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 6.30, 6.44, 6.80 [01:40:34] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [01:44:20] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.92, 7.28, 6.93 [02:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [02:01:53] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp5.wikitide.net - NS RECORDS OK [02:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:46:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.27, 7.62, 7.92 [02:48:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.62, 8.00, 8.02 [02:50:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.78, 7.92, 7.99 [02:52:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.11, 7.97, 8.00 [02:54:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 5.83, 7.35, 7.78 [02:58:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.50, 7.80, 7.86 [03:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [03:01:37] RECOVERY - db21 Backups SQL on db21 is OK: FILE_AGE OK: /var/log/backup-logs/sql-backup.log is 95 seconds old and 273 bytes [03:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [04:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [05:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [06:00:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:08:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.74, 7.41, 7.97 [06:14:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.34, 7.85, 7.95 [06:16:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.53, 7.70, 7.88 [06:18:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.76, 7.84, 7.90 [06:22:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.54, 7.85, 7.89 [06:24:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.08, 7.85, 7.88 [06:30:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.14, 7.62, 7.81 [06:34:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 10.18, 8.34, 8.00 [06:44:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.61, 7.84, 7.96 [06:52:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.10, 7.69, 7.72 [07:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [07:40:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.10, 7.64, 7.95 [07:46:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 7.87, 7.94 [07:48:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.35, 7.73, 7.88 [08:00:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.51, 7.78, 7.74 [08:06:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.73, 7.72, 7.85 [08:12:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 7.88, 7.85 [08:14:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.41, 7.84, 7.84 [08:16:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.69, 8.06, 7.91 [08:32:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.94, 7.65, 7.91 [08:38:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.14, 7.86, 7.93 [09:00:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:26:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.44, 7.52, 7.97 [09:30:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [09:40:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.61, 7.67, 7.49 [09:42:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.94, 7.53, 7.45 [09:46:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.37, 7.98, 7.64 [09:48:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.64, 7.93, 7.67 [10:00:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 8.30, 7.13, 7.21 [10:00:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:06:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 7.07, 7.66, 7.50 [10:08:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 7.96, 8.01, 7.67 [10:10:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 6.81, 7.69, 7.60 [10:14:16] PROBLEM - puppet21 Current Load on puppet21 is CRITICAL: LOAD CRITICAL - total load average: 9.16, 8.12, 7.76 [11:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [11:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:36:56] PROBLEM - mw21 MediaWiki Rendering on mw21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2813 bytes in 0.060 second response time [12:37:00] PROBLEM - jobrunner21 MediaWiki Rendering on jobrunner21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 863 bytes in 0.072 second response time [12:37:10] PROBLEM - mw22 MediaWiki Rendering on mw22 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 863 bytes in 0.062 second response time [12:39:01] PROBLEM - matomo21 HTTPS on matomo21 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 426 bytes in 0.032 second response time [12:39:18] PROBLEM - db21 MariaDB Connections on db21 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db21.wikitide.n...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_c [12:39:18] line 47Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db21.wikitide.n...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47 [12:39:47] PROBLEM - phorge21 phorge-static.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [12:39:57] PROBLEM - db21 MariaDB on db21 is CRITICAL: Can't connect to server on 'db21.wikitide.net' (115) [12:40:12] PROBLEM - phorge21 issue-tracker.wikitide.org HTTPS on phorge21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4328 bytes in 0.018 second response time [12:40:16] PROBLEM - puppet21 Current Load on puppet21 is WARNING: LOAD WARNING - total load average: 3.71, 6.05, 7.65 [12:42:11] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [12:42:49] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 54% [12:44:48] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 33% [12:46:04] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 24% [12:46:16] RECOVERY - puppet21 Current Load on puppet21 is OK: LOAD OK - total load average: 4.33, 4.99, 6.63 [12:53:53] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [12:55:49] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 40% [12:57:46] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [12:59:43] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 53% [13:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [13:01:39] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [13:03:36] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 55% [13:13:19] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [13:15:16] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 16% [13:19:15] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [13:24:00] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.12, 3.44, 3.98 [13:30:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [13:31:15] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 37% [13:32:12] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 23% [13:38:10] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [13:42:00] RECOVERY - mem21 Current Load on mem21 is OK: LOAD OK - total load average: 2.79, 3.12, 3.38 [13:44:51] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+24/-23/±22] 13https://github.com/WikiTideOrg/puppet/compare/a79e3528e33e...ff511ab36a50 [13:44:52] [02WikiTideOrg/puppet] 07AgentIsai 03ff511ab - Update back to cloud1 [13:46:30] [02WikiTideOrg/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/06132ea24696...8d23b5a5cb9a [13:46:31] [02WikiTideOrg/dns] 07AgentIsai 038d23b5a - Add bast1-public [13:50:00] PROBLEM - mem21 Current Load on mem21 is WARNING: LOAD WARNING - total load average: 3.64, 3.39, 3.36 [13:51:47] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 10% [13:52:00] RECOVERY - mem21 Current Load on mem21 is OK: LOAD OK - total load average: 2.82, 3.20, 3.30 [13:54:57] PROBLEM - db21 MariaDB on db21 is UNKNOWN: [13:56:55] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 52% [13:57:21] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [13:57:48] PROBLEM - mail21 HTTPS on mail21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [13:58:00] PROBLEM - mail21 IMAP on mail21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:10] PROBLEM - mail21 webmail.wikitide.net HTTPS on mail21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:49] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [13:59:18] PROBLEM - mail21 SMTP on mail21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:19] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 58% [14:00:44] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 52% [14:01:35] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw21 [14:02:38] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 11% [14:03:08] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw21 [14:03:18] PROBLEM - mw21 HTTPS on mw21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [14:05:15] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 39% [14:05:28] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 41% [14:06:52] PROBLEM - mem21 memcached on mem21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:25] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 93% [14:08:19] PROBLEM - puppet21 HTTPS on puppet21 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [14:08:21] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 46% [14:08:29] RECOVERY - puppet21 Puppet on puppet21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:09:22] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 19% [14:10:01] PROBLEM - puppet21 puppetserver on puppet21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:10] PROBLEM - puppet21 puppetdb on puppet21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:34] PROBLEM - bast21 Puppet on bast21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:14:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [14:15:20] PROBLEM - os21 Puppet on os21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:36] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:16:12] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 39% [14:17:15] PROBLEM - jobchron21 Puppet on jobchron21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:17:41] PROBLEM - swiftproxy21 Puppet on swiftproxy21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:19:08] PROBLEM - phorge21 Puppet on phorge21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:19:45] PROBLEM - mon21 Puppet on mon21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:21:22] PROBLEM - ldap21 Puppet on ldap21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:21:38] PROBLEM - cloud4 Puppet on cloud4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:23:01] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 46% [14:23:29] PROBLEM - matomo21 Puppet on matomo21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:24:26] PROBLEM - jobrunner21 Puppet on jobrunner21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:24:30] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:24:55] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 33% [14:25:28] PROBLEM - mw22 Puppet on mw22 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:25:40] PROBLEM - db21 Puppet on db21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:25:47] PROBLEM - os21 Current Load on os21 is WARNING: LOAD WARNING - total load average: 1.36, 1.60, 1.98 [14:26:42] PROBLEM - ldap21 Current Load on ldap21 is WARNING: LOAD WARNING - total load average: 1.54, 1.62, 1.97 [14:27:37] PROBLEM - mail21 Puppet on mail21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:28:52] PROBLEM - jobchron21 Current Load on jobchron21 is WARNING: LOAD WARNING - total load average: 1.57, 1.62, 1.93 [14:29:09] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 51% [14:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:32:28] PROBLEM - mw21 Puppet on mw21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:33:09] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 17% [14:35:24] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [14:37:16] PROBLEM - mem21 Puppet on mem21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:37:18] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 29% [14:38:02] PROBLEM - bots21 Puppet on bots21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:39:48] PROBLEM - os21 Current Load on os21 is CRITICAL: LOAD CRITICAL - total load average: 2.17, 1.80, 1.81 [14:40:11] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 42% [14:40:29] PROBLEM - puppet21 Puppet on puppet21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:40:42] PROBLEM - ldap21 Current Load on ldap21 is CRITICAL: LOAD CRITICAL - total load average: 2.15, 1.83, 1.82 [14:40:52] PROBLEM - www.farthestfrontier.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['cleo.ns.cloudflare.com.', 'may.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [14:40:52] RECOVERY - Host mw1 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [14:40:52] PROBLEM - cp5 conntrack_table_size on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [14:40:52] RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Sat 20 Jan 2024 06:52:33 AM GMT +0000. [14:40:52] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2973 bytes in 1.258 second response time [14:40:52] PROBLEM - jobchron21 Current Load on jobchron21 is CRITICAL: LOAD CRITICAL - total load average: 2.37, 1.88, 1.84 [14:40:53] PROBLEM - swiftproxy1 HTTPS on swiftproxy1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [14:40:53] PROBLEM - cp4 APT on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [14:40:55] RECOVERY - Host cp1 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [14:40:56] PROBLEM - mon1 NTP time on mon1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.111: Connection reset by peer [14:40:56] RECOVERY - test1 SSH on test1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [14:40:59] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2974 bytes in 1.990 second response time [14:40:59] RECOVERY - Host phorge1 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [14:40:59] PROBLEM - cp4 ferm_active on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [14:41:00] PROBLEM - cp2 conntrack_table_size on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [14:41:01] RECOVERY - mw1 nutcracker process on mw1 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name 'nutcracker' [14:41:03] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:41:03] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.348 second response time [14:41:06] PROBLEM - cp4 conntrack_table_size on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [14:41:06] PROBLEM - phorge1 Current Load on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:41:06] RECOVERY - test1 nutcracker port on test1 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [14:41:06] RECOVERY - mail1 HTTPS on mail1 is OK: HTTP OK: HTTP/2 301 - 227 bytes in 0.006 second response time [14:41:08] RECOVERY - jobrunner1 HTTPS on jobrunner1 is OK: HTTP OK: HTTP/2 200 - 369 bytes in 0.016 second response time [14:41:09] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.79.55.151: Connection reset by peer [14:41:09] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [14:41:10] PROBLEM - cp4 Nginx Backend for mw2 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 146.59.44.171: Connection reset by peer [14:41:10] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [14:41:10] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:41:11] PROBLEM - phorge1 php-fpm on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:41:12] PROBLEM - cp5 Nginx Backend for puppet1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [14:41:16] PROBLEM - phorge1 phd on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:41:16] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/2 200 - 364 bytes in 0.058 second response time [14:41:17] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [14:41:18] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2973 bytes in 0.612 second response time [14:41:18] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.079 seconds response time. wikitide.net returns 2607:5300:205:200::2aa8,51.79.55.151 [14:41:19] RECOVERY - www.wikitide.org - reverse DNS on sslhost is OK: SSL OK - www.wikitide.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK [14:41:19] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [14:41:20] RECOVERY - mon1 monitoring.wikitide.net HTTPS on mon1 is OK: HTTP OK: HTTP/1.1 302 Found - 298 bytes in 0.089 second response time [14:41:21] PROBLEM - phorge1 Backups Phorge Static on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:41:21] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2973 bytes in 0.225 second response time [14:41:21] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp2.wikitide.net - NS RECORDS OK [14:41:21] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [14:41:22] RECOVERY - Host matomo1 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [14:41:22] PROBLEM - cp3 GDNSD Datacenters on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.75.170.66: Connection reset by peer [14:41:24] PROBLEM - mon1 IRCEcho on mon1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.111: Connection reset by peer [14:41:24] RECOVERY - cp3 Auth DNS on cp3 is OK: DNS OK: 0.232 seconds response time. wikitide.net returns 2607:5300:205:200::2aa8,51.79.55.151 [14:41:25] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 139.99.236.151: Connection reset by peer [14:41:26] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.139 second response time [14:41:26] RECOVERY - puppet1 puppetserver on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8140 [14:41:27] RECOVERY - matomo1 conntrack_table_size on matomo1 is OK: OK: nf_conntrack is 0 % full [14:41:27] PROBLEM - cp5 NTP time on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.167.159: Connection reset by peer [14:41:28] PROBLEM - puppet1 HTTPS on puppet1 is WARNING: HTTP WARNING: HTTP/2 403 - 113 bytes in 0.012 second response time [14:41:29] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2974 bytes in 0.717 second response time [14:41:29] RECOVERY - mon1 grafana.wikitide.net HTTPS on mon1 is OK: HTTP OK: HTTP/1.1 200 OK - 43178 bytes in 0.028 second response time [14:41:31] RECOVERY - test1 NTP time on test1 is OK: NTP OK: Offset 0.004355311394 secs [14:41:32] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:41:33] PROBLEM - mem1 ferm_active on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:41:36] RECOVERY - test1 conntrack_table_size on test1 is OK: OK: nf_conntrack is 0 % full [14:41:40] PROBLEM - mem1 Disk Space on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:41:51] PROBLEM - bots1 APT on bots1 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates). [14:41:53] PROBLEM - swiftac1 conntrack_table_size on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:41:57] PROBLEM - prometheus1 Disk Space on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:41:59] PROBLEM - services1 NTP time on services1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.115: Connection reset by peer [14:42:02] PROBLEM - prometheus1 PowerDNS Recursor on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:42:09] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 24% [14:42:13] PROBLEM - services1 conntrack_table_size on services1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.115: Connection reset by peer [14:42:22] PROBLEM - graylog1 APT on graylog1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.117: Connection reset by peer [14:42:25] PROBLEM - mem1 NTP time on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:42:25] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 50 packages available for upgrade (3 critical updates). [14:42:26] PROBLEM - puppet1 NTP time on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:42:26] PROBLEM - puppet1 Backups SSLKeys on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:42:31] PROBLEM - swiftac1 APT on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:42:31] RECOVERY - test1 nutcracker process on test1 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name 'nutcracker' [14:42:31] RECOVERY - matomo1 PowerDNS Recursor on matomo1 is OK: DNS OK: 0.029 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [14:42:32] PROBLEM - mem1 APT on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:42:34] PROBLEM - services1 PowerDNS Recursor on services1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.115: Connection reset by peer [14:42:34] PROBLEM - puppet1 PowerDNS Recursor on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:42:35] PROBLEM - swiftac1 PowerDNS Recursor on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:42:36] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.441 second response time RECOVERY - mw2 SSH on mw2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [14:42:36] RECOVERY - mw1 PowerDNS Recursor on mw1 is OK: DNS OK: 0.033 seconds response time. wikitide.org returns 2607:5300:205:200::2aa8,51.79.55.151 [14:42:36] RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 8 processes with command name 'php-fpm8.2' [14:42:36] RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 17792MiB (44% inode=67%); [14:42:37] PROBLEM - swiftac1 ferm_active on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:42:38] PROBLEM - mon1 Disk Space on mon1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.111: Connection reset by peer [14:42:38] PROBLEM - puppet1 Current Load on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:42:39] PROBLEM - swiftac1 NTP time on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:42:40] PROBLEM - swiftac1 Disk Space on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:42:40] PROBLEM - os1 APT on os1 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [14:42:41] PROBLEM - swiftproxy1 APT on swiftproxy1 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (3 critical updates). [14:42:41] RECOVERY - phorge1 issue-tracker.wikitide.org HTTPS on phorge1 is OK: HTTP OK: HTTP/1.1 200 OK - 110931 bytes in 0.143 second response time [14:42:41] PROBLEM - phorge1 Disk Space on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:42:41] RECOVERY - test1 ferm_active on test1 is OK: OK ferm input default policy is set [14:42:41] PROBLEM - phorge1 conntrack_table_size on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:42:42] PROBLEM - graylog1 Current Load on graylog1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.117: Connection reset by peer [14:42:43] PROBLEM - mem1 PowerDNS Recursor on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:42:46] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.106 second response time [14:42:48] PROBLEM - mem1 conntrack_table_size on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:42:51] PROBLEM - phorge1 PowerDNS Recursor on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:42:51] PROBLEM - phorge1 APT on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:42:51] PROBLEM - phorge1 NTP time on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer PROBLEM - phorge1 ferm_active on phorge1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.108: Connection reset by peer [14:42:52] PROBLEM - graylog1 NTP time on graylog1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.117: Connection reset by peer [14:42:52] PROBLEM - bast1 APT on bast1 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates). [14:42:55] PROBLEM - mon1 ferm_active on mon1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.111: Connection reset by peer [14:42:55] PROBLEM - puppet1 Backups Private on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:42:56] PROBLEM - prometheus1 conntrack_table_size on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:42:57] PROBLEM - puppet1 ferm_active on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:42:59] PROBLEM - mon1 php-fpm on mon1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.111: Connection reset by peer [14:43:01] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 46% [14:43:02] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (3 critical updates). [14:43:04] PROBLEM - prometheus1 APT on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:43:05] PROBLEM - ldap1 APT on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:43:07] PROBLEM - prometheus1 ferm_active on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:43:07] PROBLEM - prometheus1 Current Load on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:43:07] PROBLEM - graylog1 ferm_active on graylog1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.117: Connection reset by peer [14:43:08] PROBLEM - mem1 Current Load on mem1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.106: Connection reset by peer [14:43:09] PROBLEM - services1 ferm_active on services1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.115: Connection reset by peer [14:43:11] PROBLEM - prometheus1 NTP time on prometheus1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.118: Connection reset by peer [14:43:13] PROBLEM - services1 Disk Space on services1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.115: Connection reset by peer [14:43:13] PROBLEM - swiftac1 Current Load on swiftac1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.120: Connection reset by peer [14:43:15] PROBLEM - services1 Current Load on services1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.115: Connection reset by peer [14:43:16] RECOVERY - matomo1 SSH on matomo1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [14:43:16] PROBLEM - puppet1 conntrack_table_size on puppet1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.100: Connection reset by peer [14:43:16] PROBLEM - swiftobject1 APT on swiftobject1 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (3 critical updates). [14:43:18] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.117: Connection reset by peer [14:43:21] PROBLEM - graylog1 conntrack_table_size on graylog1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.117: Connection reset by peer [14:43:22] PROBLEM - db1 APT on db1 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (3 critical updates). [14:43:26] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (3 critical updates). [14:44:47] PROBLEM - ldap1 Current Load on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:44:55] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [14:45:00] PROBLEM - ldap1 NTP time on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:45:01] PROBLEM - ldap1 Disk Space on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:45:07] PROBLEM - ldap1 conntrack_table_size on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:45:19] PROBLEM - ldap1 PowerDNS Recursor on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:45:24] PROBLEM - ldap1 ferm_active on ldap1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.109: Connection reset by peer [14:46:10] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [14:46:24] PROBLEM - swiftobject1 ferm_active on swiftobject1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.121: Connection reset by peer [14:46:31] PROBLEM - mail1 php-fpm on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:46:37] PROBLEM - mail1 PowerDNS Recursor on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:46:43] PROBLEM - mail1 ferm_active on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:46:49] PROBLEM - swiftobject1 conntrack_table_size on swiftobject1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.121: Connection reset by peer [14:46:49] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 39% [14:46:56] PROBLEM - mail1 NTP time on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:46:58] PROBLEM - mail1 conntrack_table_size on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:46:59] PROBLEM - mail1 Disk Space on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:46:59] PROBLEM - swiftobject1 PowerDNS Recursor on swiftobject1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.121: Connection reset by peer [14:47:02] PROBLEM - swiftobject1 NTP time on swiftobject1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.121: Connection reset by peer [14:47:02] PROBLEM - swiftobject1 Disk Space on swiftobject1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.121: Connection reset by peer [14:47:09] PROBLEM - swiftobject1 Current Load on swiftobject1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.121: Connection reset by peer [14:47:23] PROBLEM - mail1 Current Load on mail1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.110: Connection reset by peer [14:48:08] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 38% [14:49:01] PROBLEM - jobchron1 Disk Space on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:49:09] PROBLEM - jobchron1 Current Load on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:49:20] PROBLEM - jobchron1 PowerDNS Recursor on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:49:34] PROBLEM - jobchron1 NTP time on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:50:32] PROBLEM - bots1 conntrack_table_size on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:50:37] PROBLEM - jobchron1 conntrack_table_size on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:50:43] PROBLEM - jobchron1 JobChron Service on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:50:46] PROBLEM - bots1 Disk Space on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:50:51] PROBLEM - jobchron1 ferm_active on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:50:54] PROBLEM - bots1 PowerDNS Recursor on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:50:57] PROBLEM - jobchron1 poolcounter process on jobchron1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.105: Connection reset by peer [14:51:01] PROBLEM - bots1 Current Load on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:51:03] PROBLEM - bots1 ferm_active on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:51:09] PROBLEM - bots1 IRC Log Bot on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:51:26] PROBLEM - bots1 NTP time on bots1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.123: Connection reset by peer [14:52:09] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [14:52:14] PROBLEM - db1 Backups SQL on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:52:17] PROBLEM - matomo1 PowerDNS Recursor on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:52:19] PROBLEM - db1 Backups SQL wtglobal on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:52:33] PROBLEM - db1 conntrack_table_size on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:52:34] PROBLEM - matomo1 php-fpm on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:52:38] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet last ran 1 hour ago [14:52:40] PROBLEM - db1 ferm_active on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:52:43] PROBLEM - db1 PowerDNS Recursor on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:52:50] PROBLEM - matomo1 HTTPS on matomo1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:52:57] PROBLEM - matomo1 ferm_active on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:52:58] PROBLEM - matomo1 Disk Space on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:53:07] PROBLEM - matomo1 NTP time on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:53:08] PROBLEM - matomo1 Current Load on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:53:09] PROBLEM - db1 Current Load on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:53:09] PROBLEM - matomo1 conntrack_table_size on matomo1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.119: Connection reset by peer [14:53:15] PROBLEM - db1 Disk Space on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:53:18] PROBLEM - db1 NTP time on db1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.103: Connection reset by peer [14:53:29] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet last ran 1 hour ago [14:53:48] PROBLEM - os21 Current Load on os21 is WARNING: LOAD WARNING - total load average: 1.51, 1.98, 1.99 [14:54:16] PROBLEM - bast1 Disk Space on bast1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.114: Connection reset by peer [14:54:18] PROBLEM - bast1 ferm_active on bast1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.114: Connection reset by peer [14:54:18] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 864 bytes in 0.204 second response time [14:54:25] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:30] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 63.141.240.4: Connection reset by peer [14:54:36] PROBLEM - bast1 PowerDNS Recursor on bast1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.114: Connection reset by peer [14:54:41] PROBLEM - bast1 NTP time on bast1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.114: Connection reset by peer [14:54:41] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 63.141.240.4: Connection reset by peer [14:54:43] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 63.141.240.4: Connection reset by peer [14:54:45] PROBLEM - bast1 Current Load on bast1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.114: Connection reset by peer [14:54:49] PROBLEM - bast1 conntrack_table_size on bast1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.114: Connection reset by peer [14:54:52] PROBLEM - jobchron21 Current Load on jobchron21 is WARNING: LOAD WARNING - total load average: 1.78, 2.00, 2.00 [14:55:10] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 63.141.240.4: Connection reset by peer [14:55:11] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 63.141.240.4: Connection reset by peer [14:55:52] PROBLEM - cp5 Puppet on cp5 is WARNING: WARNING: Puppet last ran 1 hour ago [14:56:06] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 49% [14:56:14] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.114 second response time [14:56:42] PROBLEM - ldap21 Current Load on ldap21 is WARNING: LOAD WARNING - total load average: 1.70, 1.90, 1.96 [14:56:50] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.911 second response time [14:57:23] PROBLEM - os1 Disk Space on os1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.116: Connection reset by peer [14:57:47] PROBLEM - os21 Current Load on os21 is CRITICAL: LOAD CRITICAL - total load average: 2.56, 2.08, 2.02 [14:58:00] PROBLEM - os1 PowerDNS Recursor on os1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.116: Connection reset by peer [14:58:20] PROBLEM - os1 conntrack_table_size on os1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.116: Connection reset by peer [14:58:28] PROBLEM - os1 ferm_active on os1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.116: Connection reset by peer [14:58:35] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 44% [14:58:42] PROBLEM - ldap21 Current Load on ldap21 is CRITICAL: LOAD CRITICAL - total load average: 2.44, 2.14, 2.04 [14:58:43] PROBLEM - os1 Current Load on os1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.116: Connection reset by peer [14:58:44] PROBLEM - os1 HTTPS on os1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [14:58:52] PROBLEM - jobchron21 Current Load on jobchron21 is CRITICAL: LOAD CRITICAL - total load average: 2.39, 2.14, 2.04 [14:59:29] PROBLEM - os1 NTP time on os1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.116: Connection reset by peer [15:00:33] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [15:01:24] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:58] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [15:02:24] PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.122: Connection reset by peer [15:02:26] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.122: Connection reset by peer [15:02:37] PROBLEM - swiftproxy1 Disk Space on swiftproxy1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.122: Connection reset by peer [15:02:46] PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.122: Connection reset by peer [15:03:07] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.122: Connection reset by peer [15:03:15] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.0.0.122: Connection reset by peer [15:03:21] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.111 second response time [15:03:56] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 29% [15:06:35] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 37% [15:10:06] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 45% [15:12:03] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 39% [15:13:35] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [15:15:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 40% [15:17:24] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [15:21:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 53% [15:23:06] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [15:30:34] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [15:30:43] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 32% [15:33:33] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 42% [15:37:49] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [15:39:30] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 39% [15:39:48]