[00:00:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.30, 21.74, 23.63 [00:02:48] PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query. [00:03:49] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.47, 3.49, 3.91 [00:05:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.38, 4.30, 4.16 [00:06:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 22.57, 23.41 [00:06:30] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:06:36] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:08:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.35, 21.47, 22.91 [00:08:38] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.794 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:11:48] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.51, 3.72, 3.98 [00:13:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.28, 4.79, 4.35 [00:18:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.21, 17.65, 20.25 [00:19:10] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:21:06] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.117 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:24:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.05, 21.48, 20.98 [00:28:10] RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:29:38] [02mw-config] 07hurohukidaikon reviewed pull request 03#5641 commit - 13https://github.com/miraheze/mw-config/pull/5641#discussion_r1725991555 [00:29:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.17, 3.46, 3.96 [00:31:30] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:31:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 3.82, 4.02 [00:32:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.17, 22.92, 22.05 [00:41:48] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.13, 3.34, 3.94 [00:43:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.90, 3.69, 4.01 [00:44:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.22, 23.84, 22.81 [00:46:43] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:46:49] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:56] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.114 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:55:11] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [00:56:21] RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:56:30] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:00:17] PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query. [01:01:30] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:04:04] [02mw-config] 07Universal-Omega reviewed pull request 03#5641 commit - 13https://github.com/miraheze/mw-config/pull/5641#discussion_r1726008155 [01:05:21] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:07:17] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.071 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:08:30] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query. [01:11:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.62, 3.47, 3.95 [01:12:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.29, 22.24, 23.81 [01:16:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 24.52, 24.31 [01:19:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.45, 2.66, 3.35 [01:20:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.62, 23.42, 23.96 [01:25:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.83, 4.13, 3.63 [01:26:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 23.66, 23.89 [01:26:43] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:28:39] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:30:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.94, 23.22, 23.83 [01:31:30] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:32:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 25.59, 24.63 [01:37:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.90, 3.49, 3.75 [01:38:01] RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:39:48] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.64, 3.77 [01:40:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.70, 23.21, 23.92 [01:41:17] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:41:55] [02mw-config] 07hurohukidaikon synchronize pull request 03#5641: Add UploadWizard configurations for kagagawiki in LocalWiki.php - 13https://github.com/miraheze/mw-config/pull/5641 [01:42:47] [02mw-config] 07hurohukidaikon reviewed pull request 03#5641 commit - 13https://github.com/miraheze/mw-config/pull/5641#discussion_r1726120982 [01:43:21] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.145 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:44:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.18, 25.65, 24.68 [01:45:09] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:46:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.73, 23.11, 23.89 [01:50:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 23.64, 23.91 [01:51:30] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:52:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.67, 23.25, 23.80 [01:52:52] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:53:18] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [01:54:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.16, 23.25, 23.67 [01:56:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.32, 22.50, 23.36 [01:56:55] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.071 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:58:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.68, 25.00, 24.18 [02:00:20] RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:00:23] RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:01:30] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:02:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.25, 22.90, 23.62 [02:02:44] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:06:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:07:53] RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:09:31] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query. [02:10:45] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:11:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:12:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 23.27, 23.14 [02:12:41] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.212 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [02:13:43] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [02:14:08] RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:16:53] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:50] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [02:25:20] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:33:33] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [02:33:37] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.63, 19.51, 17.93 [02:35:31] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.80, 18.38, 17.70 [02:40:39] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:43:55] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.83, 22.35, 19.73 [02:44:05] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.98, 21.92, 19.46 [02:44:44] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [02:45:59] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.85, 20.08, 19.07 [02:46:10] PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query. [02:47:43] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.28, 19.17, 19.02 [02:47:48] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:51:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:52:24] miraheze/mw-config - hurohukidaikon the build passed. [02:52:48] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [02:54:04] [02mw-config] 07Universal-Omega synchronize pull request 03#5641: Add UploadWizard configurations for kagagawiki in LocalWiki.php - 13https://github.com/miraheze/mw-config/pull/5641 [02:54:57] miraheze/mw-config - Universal-Omega the build passed. [02:55:21] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'} [02:55:39] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7d4818bc93c7...245eb70a3b11 [02:55:40] [02mw-config] 07hurohukidaikon 03245eb70 - Add UploadWizard configurations for kagagawiki in LocalWiki.php (#5641) [02:55:43] [02mw-config] 07Universal-Omega closed pull request 03#5641: Add UploadWizard configurations for kagagawiki in LocalWiki.php - 13https://github.com/miraheze/mw-config/pull/5641 [02:56:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:56:32] miraheze/mw-config - Universal-Omega the build passed. [02:56:41] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:00:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.61, 22.54, 23.98 [03:00:48] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 5.449 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:12:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 21.09, 21.97 [03:13:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.26, 2.67, 3.80 [03:14:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.72, 20.02, 21.49 [03:15:20] RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:16:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.40, 21.57, 21.85 [03:16:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:17:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 3.51, 3.87 [03:18:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.81, 21.18, 21.68 [03:22:34] RECOVERY - wiki.gowashtenaw.org - reverse DNS on sslhost is OK: SSL OK - wiki.gowashtenaw.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:23:43] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:26:08] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:26:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:28:35] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.08, 22.62, 21.89 [03:30:33] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [03:31:07] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [03:31:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:32:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.63, 22.38, 21.91 [03:36:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 22.32, 21.90 [03:36:07] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.077 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:38:00] RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:38:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.22, 22.07, 21.83 [03:40:30] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:42:26] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.099 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:46:35] RECOVERY - guia.esporo.net - reverse DNS on sslhost is OK: SSL OK - guia.esporo.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:48:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.10, 24.14, 22.31 [03:48:54] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.39, 21.16, 18.41 [03:49:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.18, 20.68, 18.42 [03:50:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.38, 23.73, 22.45 [03:50:48] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.92, 18.84, 17.86 [03:51:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.25, 18.55, 17.88 [03:52:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 23.31, 22.39 [03:53:45] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:54:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.70, 22.60, 22.30 [03:56:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 23.18, 22.52 [03:56:09] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:56:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:57:54] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 7.880 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [03:58:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.02, 22.78, 22.48 [03:59:32] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:01:30] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:04:08] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [04:06:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 21.75, 21.71 [04:06:41] PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query. [04:07:50] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.94, 22.51, 22.07 [04:09:54] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.minkyu.kim All nameservers failed to answer the query. [04:14:03] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:16:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.79, 22.29, 21.76 [04:16:10] PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - unhingedwiki.paralleldv.org All nameservers failed to answer the query. [04:18:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.22, 22.18, 21.79 [04:23:43] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:24:06] PROBLEM - wiki.gowashtenaw.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gowashtenaw.org All nameservers failed to answer the query. [04:25:20] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query. [04:25:39] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.302 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:26:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:28:15] PROBLEM - wiki.orivium.io - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.orivium.io All nameservers failed to answer the query. [04:31:19] PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eastsidegamers.no All nameservers failed to answer the query. [04:31:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:37:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.64, 3.08, 3.85 [04:39:02] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query. [04:40:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.07, 19.47, 20.36 [04:43:46] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 3.58, 3.75 [04:47:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.67, 3.17, 3.54 [04:49:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.38, 2.91, 3.40 [04:52:52] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:28] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:53:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.98, 5.49, 4.36 [04:54:30] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.85, 21.63, 17.68 [04:54:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.97, 20.37, 17.14 [04:54:44] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.07, 21.20, 17.52 [04:54:49] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:54:54] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw181 [04:55:05] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw181 mw182 [04:55:08] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.66, 17.42, 13.74 [04:55:12] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.90, 21.10, 17.18 [04:55:19] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [04:55:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 20.63, 16.43 [04:55:37] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.40, 24.07, 21.33 [04:56:30] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.70, 21.89, 18.31 [04:56:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.22, 20.60, 17.66 [04:56:44] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.05, 20.59, 17.76 [04:56:54] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [04:57:05] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [04:57:08] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.81, 15.61, 13.53 [04:57:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.37, 18.43, 16.71 [04:57:19] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [04:57:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.36, 18.32, 16.06 [04:57:34] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.11, 23.50, 21.46 [04:58:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.64, 20.02, 17.89 [04:59:31] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.03, 25.16, 22.33 [05:00:30] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.61, 19.17, 18.04 [05:00:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.33, 18.93, 17.75 [05:01:07] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:01:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:01:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.15, 23.59, 22.09 [05:02:21] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:10] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [05:03:55] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 8.497 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:04:18] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [05:06:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:13:08] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.85, 18.61, 20.17 [05:17:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.93, 2.66, 3.77 [05:21:47] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.52, 2.33, 3.40 [05:25:35] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:25:47] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.09, 5.02, 4.22 [05:27:23] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:29:38] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.565 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:31:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:35:36] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 20 minutes ago with 0 failures [05:36:05] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:36:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:41:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:42:12] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.166 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:42:16] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.05, 19.97, 18.86 [05:43:12] PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864161 seconds old and 48966 bytes [05:44:13] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.09, 19.26, 18.71 [05:46:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:51:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:53:12] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.85, 20.23, 16.48 [05:53:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.93, 20.26, 15.33 [05:54:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.71, 23.43, 20.08 [05:55:12] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.32, 20.90, 17.25 [05:55:32] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.02, 19.28, 15.57 [05:56:20] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:57:12] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.49, 19.41, 17.15 [05:58:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.42, 23.30, 20.80 [05:58:09] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:00:04] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.246 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:02:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 22.94, 21.11 [06:04:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.74, 20.83, 20.58 [06:06:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.90, 19.35, 20.08 [06:07:53] RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: SSL OK - wiki.minkyu.kim reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:08:41] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:08:58] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:09:44] PROBLEM - wiki.wubbygame.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wubbygame.com All nameservers failed to answer the query. [06:09:46] PROBLEM - swests.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - swests.org All nameservers failed to answer the query. [06:10:33] PROBLEM - www.dovearchives.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.dovearchives.wiki All nameservers failed to answer the query. [06:10:36] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.080 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:10:55] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [06:14:08] RECOVERY - unhingedwiki.paralleldv.org - reverse DNS on sslhost is OK: SSL OK - unhingedwiki.paralleldv.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:14:15] PROBLEM - www.istpcomputing.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.istpcomputing.com All nameservers failed to answer the query. [06:14:44] PROBLEM - wiki.gamergeeked.us - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gamergeeked.us All nameservers failed to answer the query. [06:20:48] PROBLEM - wiki.scvo.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.scvo.org All nameservers failed to answer the query. [06:21:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:23:07] RECOVERY - wiki.gowashtenaw.org - reverse DNS on sslhost is OK: SSL OK - wiki.gowashtenaw.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:24:05] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.triplescripts.org All nameservers failed to answer the query. [06:25:19] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'} [06:26:04] RECOVERY - wiki.orivium.io - reverse DNS on sslhost is OK: SSL OK - wiki.orivium.io reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:28:52] RECOVERY - wiki.eastsidegamers.no - reverse DNS on sslhost is OK: SSL OK - wiki.eastsidegamers.no reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:31:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:33:26] PROBLEM - wiki.18t.rip - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.18t.rip All nameservers failed to answer the query. [06:33:29] PROBLEM - wiki.sadboyzpod.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sadboyzpod.com All nameservers failed to answer the query. [06:33:46] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.54, 2.75, 3.84 [06:34:30] PROBLEM - zh.tardis.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zh.tardis.wiki All nameservers failed to answer the query. [06:35:51] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.chevrine.com All nameservers failed to answer the query. [06:36:50] RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:37:01] RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:37:57] PROBLEM - www.samafia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.samafia.org All nameservers failed to answer the query. [06:39:46] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.58, 2.21, 3.26 [06:40:07] RECOVERY - www.dovearchives.wiki - reverse DNS on sslhost is OK: SSL OK - www.dovearchives.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:43:54] RECOVERY - wiki.gamergeeked.us - reverse DNS on sslhost is OK: SSL OK - wiki.gamergeeked.us reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:46:27] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o