[00:02:23] RECOVERY - cp1 Disk Space on cp1 is OK: DISK OK - free space: / 6139MiB (11% inode=97%); [00:06:28] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [00:28:36] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [00:37:16] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [01:06:23] PROBLEM - cp1 Disk Space on cp1 is WARNING: DISK WARNING - free space: / 6071MiB (10% inode=97%); [01:06:24] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [01:30:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:28:38] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:38:20] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [02:38:30] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [03:00:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:07:02] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:36:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [03:41:31] [02WikiTideOrg/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/8f46b57909dc...763214ec41f4 [03:41:32] [02WikiTideOrg/dns] 07AgentIsai 03763214e - Add bast21 public [04:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [04:16:16] PROBLEM - db1 Backups SQL on db1 is CRITICAL: FILE_AGE CRITICAL: /var/log/backup-logs/sql-backup.log is 1209689 seconds old and 1977 bytes [04:36:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [04:39:08] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6c76efdc8392...200983e80b02 [04:39:09] [02WikiTideOrg/puppet] 07AgentIsai 03200983e - Use bots21 for logsalmsg [04:43:02] PROBLEM - bots1 IRC-Discord Relay Bot on bots1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'relaybot' [04:58:38] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [05:02:59] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±16] 13https://github.com/WikiTideOrg/puppet/compare/200983e80b02...ada86b1c0992 [05:03:01] [02WikiTideOrg/puppet] 07AgentIsai 03ada86b1 - Update more hostnames [05:05:41] !log [agent@mon21] test [05:05:43] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:06:06] !log [agent@mon21] test2 [05:06:08] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:18:49] PROBLEM - bots1 IRC Limnoria Bot on bots1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'limnoria' [05:30:46] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:04:38] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:36:26] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [10:58:36] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [11:30:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:25:13] PROBLEM - os1 Disk Space on os1 is CRITICAL: DISK CRITICAL - free space: / 11256MiB (5% inode=99%); [14:31:13] PROBLEM - os1 Disk Space on os1 is WARNING: DISK WARNING - free space: / 11685MiB (6% inode=99%); [14:41:12] PROBLEM - os1 Disk Space on os1 is CRITICAL: DISK CRITICAL - free space: / 11283MiB (5% inode=99%); [14:45:12] PROBLEM - os1 Disk Space on os1 is WARNING: DISK WARNING - free space: / 11663MiB (6% inode=99%); [14:49:12] PROBLEM - os1 Disk Space on os1 is CRITICAL: DISK CRITICAL - free space: / 11273MiB (5% inode=99%); [14:55:13] PROBLEM - os1 Disk Space on os1 is WARNING: DISK WARNING - free space: / 11641MiB (6% inode=99%); [15:08:23] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [15:09:26] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [15:14:25] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [15:15:13] PROBLEM - os1 Disk Space on os1 is CRITICAL: DISK CRITICAL - free space: / 11282MiB (5% inode=99%); [15:28:38] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [15:37:58] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [15:41:35] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [15:43:12] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [16:07:13] PROBLEM - os1 Disk Space on os1 is WARNING: DISK WARNING - free space: / 12430MiB (6% inode=99%); [16:07:19] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [16:07:30] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [16:11:19] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [16:15:01] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/MediaWikiDebugJS/compare/143f911a9a97...24d337bf8bc0 [16:15:02] [02WikiTideOrg/MediaWikiDebugJS] 07Universal-Omega 0324d337b - Include fallback URL in console.log [16:30:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:42:14] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:06:25] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [17:32:01] PROBLEM - mw2 NTP time on mw2 is WARNING: NTP WARNING: Offset 0.1357597709 secs [17:42:01] RECOVERY - mw2 NTP time on mw2 is OK: NTP OK: Offset -0.03180706501 secs [17:45:44] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/ada86b1c0992...44eb751e140e [17:45:47] [02WikiTideOrg/puppet] 07AgentIsai 0344eb751 - Update hostnames [18:06:27] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [18:14:22] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query. [18:23:38] PROBLEM - graylog1 Disk Space on graylog1 is WARNING: DISK WARNING - free space: / 766MiB (10% inode=84%); [18:39:38] PROBLEM - graylog1 Disk Space on graylog1 is CRITICAL: DISK CRITICAL - free space: / 452MiB (5% inode=84%); [19:12:18] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rosettacode.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [19:13:01] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [19:13:57] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [19:36:24] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [19:42:38] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [19:43:42] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:08:30] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [20:12:12] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:12:16] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:34:01] PROBLEM - mw2 NTP time on mw2 is WARNING: NTP WARNING: Offset -0.1088602543 secs [20:37:21] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:58:01] RECOVERY - mw2 NTP time on mw2 is OK: NTP OK: Offset -0.08979853988 secs [21:39:26] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [22:08:20] RECOVERY - mlrpgspeedruns.com - reverse DNS on sslhost is OK: SSL OK - mlrpgspeedruns.com reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:28:36] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:38:29] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mlrpgspeedruns.com. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [23:43:40] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.rosettacode.org All nameservers failed to answer the query. [23:44:22] PROBLEM - rosettacode.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - rosettacode.org All nameservers failed to answer the query.