[00:01:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:18] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [00:02:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:03:07] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:03:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:05:59] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:59] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:06:45] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:08:45] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:03] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:10:22] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:29] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:11:20] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/57eb83a9446c...195584db5f35 [00:11:22] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03195584d - Bot: Update SSL cert for realitylongterms.studio [00:11:57] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:43] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:31] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:17:04] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [00:17:13] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:19:51] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:20:30] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:21:13] RECOVERY - matomo1 Puppet on matomo1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:21:44] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:22:15] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:22:54] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:25:52] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:26:08] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:26:27] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:29:54] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [00:49:40] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:03:07] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:04:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:17:23] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:31:07] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:34:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:47:20] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [04:34:18] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [05:02:18] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [13:44:43] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:48:32] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Tue 13 Feb 2024 01:26:39 PM GMT +0000). [14:07:58] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:07:59] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:08:45] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:08:45] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:12:27] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:12:29] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:12:43] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [14:13:57] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:14:04] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:34:28] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:34:45] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:35:59] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:44:43] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:05:59] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:45] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:36:45] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:46:31] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:48:44] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:49:13] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:50:30] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:51:40] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:51:51] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:53:13] PROBLEM - matomo1 Puppet on matomo1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:53:45] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:53:54] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:54:15] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:57:52] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:58:08] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:58:28] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:01:54] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:03:07] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:06:45] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:07:58] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:07:59] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.