[00:00:43] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:43] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:13:01] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:42:59] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [03:02:39] PROBLEM - mon1 Backups Grafana on mon1 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864157 seconds old and 347 bytes [03:29:33] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [17:48:33] PROBLEM - puppet1 APT on puppet1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:50:34] PROBLEM - puppet1 APT on puppet1 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (1 critical updates). [20:12:04] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:22] PROBLEM - matomo1 PowerDNS Recursor on matomo1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:52:25] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:52:36] PROBLEM - data.lophocmatngu.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:52:36] PROBLEM - www.rosettacode.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:52:47] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:52:55] PROBLEM - mw1 PowerDNS Recursor on mw1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:53:00] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:53:04] PROBLEM - mon1 PowerDNS Recursor on mon1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:53:08] PROBLEM - phorge1 PowerDNS Recursor on phorge1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:53:35] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 933 bytes in 0.088 second response time [20:53:40] PROBLEM - cp4 PowerDNS Recursor on cp4 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:53:46] PROBLEM - puppet1 PowerDNS Recursor on puppet1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:53:54] PROBLEM - bots1 PowerDNS Recursor on bots1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:54:08] PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:54:40] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:54:46] PROBLEM - test1 PowerDNS Recursor on test1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:55:11] PROBLEM - jobchron1 PowerDNS Recursor on jobchron1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:55:13] PROBLEM - db1 MariaDB on db1 is UNKNOWN: check_mysql: Invalid hostname/address - db1.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group] [20:55:55] PROBLEM - db1 MariaDB Connections on db1 is UNKNOWN: PHP Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db1.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47Warning: mysqli_real_connect(): php_network_getaddresses: getaddrinfo for db1.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_ [20:55:55] n line 47PHP Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db1.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown i [20:55:55] /plugins/check_mysql_connections.php on line 47Fatal error: Uncaught mysqli_sql_exception: php_network_getaddresses: getaddrinfo for db1.wikitide.net failed: Temporary failure in name resolution in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NU [20:55:55] 1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47 [20:55:57] PROBLEM - jobrunner1 PowerDNS Recursor on jobrunner1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:56:39] PROBLEM - prometheus1 PowerDNS Recursor on prometheus1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:56:45] PROBLEM - mw2 PowerDNS Recursor on mw2 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:56:45] PROBLEM - projects.dmvpetridish.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:57:34] PROBLEM - ldap1 LDAP on ldap1 is CRITICAL: Could not bind to the LDAP server [20:57:48] PROBLEM - mem1 PowerDNS Recursor on mem1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:58:34] PROBLEM - os1 PowerDNS Recursor on os1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:58:50] PROBLEM - swiftac1 PowerDNS Recursor on swiftac1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:59:21] PROBLEM - graylog1 PowerDNS Recursor on graylog1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:59:23] PROBLEM - db1 PowerDNS Recursor on db1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:59:31] PROBLEM - services1 PowerDNS Recursor on services1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:59:39] PROBLEM - cp2 PowerDNS Recursor on cp2 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:59:47] PROBLEM - ldap1 PowerDNS Recursor on ldap1 is CRITICAL: Domain 'wikitide.org' was not found by the server [21:00:03] PROBLEM - bast1 PowerDNS Recursor on bast1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:00:06] PROBLEM - swiftobject1 PowerDNS Recursor on swiftobject1 is CRITICAL: Domain 'wikitide.org' was not found by the server [21:02:02] PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:02] PROBLEM - wiki.myehs.eu - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:02] PROBLEM - wiki.chevrine.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:24] PROBLEM - valorantjp.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:49] PROBLEM - wikitide.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:51] PROBLEM - nexttide.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:55] PROBLEM - mason.sagan4.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:07:34] PROBLEM - sdiy.info - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:07:43] PROBLEM - www.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:09:14] PROBLEM - www.farthestfrontier.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:09:14] PROBLEM - hsck.lophocmatngu.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:09:34] PROBLEM - meta.sagan4.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:15:15] PROBLEM - newcascadia.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:17:39] PROBLEM - wiki.burning.art - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:18:31] PROBLEM - www.greatamerica.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:18:31] PROBLEM - wiki.colleimadcat.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:18:31] PROBLEM - beta.sagan4.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:19:47] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.05, 6.89, 6.20 [21:19:47] PROBLEM - alpha.sagan4.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:19:47] PROBLEM - mlrpgspeedruns.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:20:01] PROBLEM - wikitide.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:20:16] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:21:46] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.01, 6.60, 6.18 [21:24:38] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:36] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.151 second response time [21:27:42] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 6.93, 6.85, 6.44 [21:35:37] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.49, 6.75, 6.58 [22:01:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:48] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.186 second response time [22:24:38] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:36] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.147 second response time [22:49:50] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.09, 6.93, 6.81 [22:53:48] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.54, 6.75, 6.77 [23:06:39] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 7.30, 6.99, 6.85 [23:38:38] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:36] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.150 second response time [23:58:11] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 6.07, 6.62, 6.76