[00:01:07] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [00:01:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:03:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:05:58] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:59] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:06:45] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:08:45] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:22] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:29] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:11:57] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:03] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:43] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:14:31] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [00:17:14] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:18:54] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:19:40] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:19:51] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [00:20:30] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:21:13] RECOVERY - matomo1 Puppet on matomo1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [00:21:44] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:22:14] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:01] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:25:52] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:28:08] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:28:27] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:29:54] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:24:18] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [01:32:18] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [03:08:45] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:12:11] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:12:25] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:12:29] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:13:57] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:14:43] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:34:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:37:59] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:37:59] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:38:45] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:46:31] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:49:13] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:49:32] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:51:41] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:51:51] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:52:30] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:53:13] PROBLEM - matomo1 Puppet on matomo1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:53:44] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:54:15] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:56:07] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:57:52] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:58:28] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:00:08] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:01:54] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:05:07] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:32:18] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [14:38:33] RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [14:38:51] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [14:39:58] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [14:40:14] RECOVERY - wikitide.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.com' will expire on Mon 01 Apr 2024 09:14:18 PM GMT +0000. [14:40:14] RECOVERY - farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Fri 19 Apr 2024 06:29:37 PM GMT +0000. [14:41:01] RECOVERY - distrowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'distrowiki.net' will expire on Fri 05 Apr 2024 11:46:50 PM GMT +0000. [14:41:39] RECOVERY - hsck.lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Fri 12 Apr 2024 11:08:03 PM GMT +0000. [14:43:34] RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Fri 12 Apr 2024 05:51:56 PM GMT +0000. [14:43:46] RECOVERY - www.wikitide.org - reverse DNS on sslhost is OK: SSL OK - www.wikitide.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [14:44:27] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [14:45:09] RECOVERY - projects.dmvpetridish.com - LetsEncrypt on sslhost is OK: OK - Certificate 'projects.dmvpetridish.com' will expire on Thu 29 Feb 2024 03:18:01 PM GMT +0000. [14:45:14] RECOVERY - wiki.chevrine.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chevrine.com' will expire on Thu 29 Feb 2024 03:19:26 PM GMT +0000. [14:45:23] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [14:45:24] RECOVERY - data.lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.lophocmatngu.wiki' will expire on Fri 19 Apr 2024 06:22:59 PM GMT +0000. [14:45:47] RECOVERY - realitylongterms.studio - reverse DNS on sslhost is OK: SSL OK - realitylongterms.studio reverse DNS resolves to cp3.wikitide.net - CNAME FLAT [14:45:49] RECOVERY - hkrail.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hkrail.wiki' will expire on Mon 22 Apr 2024 02:00:42 PM GMT +0000. [14:45:59] RECOVERY - valorantjp.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valorantjp.wiki' will expire on Fri 05 Apr 2024 11:17:32 PM GMT +0000. [14:46:08] RECOVERY - wiki.burning.art - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [14:46:13] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 05 Apr 2024 11:35:24 PM GMT +0000. [14:47:20] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [14:47:21] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Fri 05 Apr 2024 11:07:53 PM GMT +0000. [14:47:21] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Tue 13 Feb 2024 01:26:39 PM GMT +0000. [14:47:36] RECOVERY - wiki.myehs.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.myehs.eu' will expire on Sat 23 Mar 2024 02:44:20 AM GMT +0000. [14:48:07] RECOVERY - www.greatamerica.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'greatamerica.wiki' will expire on Fri 05 Apr 2024 11:24:34 PM GMT +0000. [19:50:38] PROBLEM - mw2 APT on mw2 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:52:39] PROBLEM - mw2 APT on mw2 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (9 critical updates). [21:15:01] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [21:18:59] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 23 day(s) (Tue 20 Feb 2024 12:20:10 PM GMT +0000). [21:35:12] PROBLEM - matomo1 HTTPS on matomo1 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.586 second response time [21:39:15] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.417 second response time [22:10:28] PROBLEM - matomo1 HTTPS on matomo1 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.007 second response time [22:14:31] RECOVERY - matomo1 HTTPS on matomo1 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.425 second response time