[00:00:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:45:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [01:25:27] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: LOAD CRITICAL - total load average: 19.55, 20.28, 9.36 [01:31:08] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.17, 6.65, 6.55 [02:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [02:45:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [03:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [03:45:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [04:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [04:19:43] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a1eb2ee4b115...e14f79527602 [04:19:44] [02WikiTideOrg/puppet] 07AgentIsai 03e14f795 - Fix swiftproxy IP [05:15:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [05:45:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [06:45:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [07:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [08:15:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:18:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [10:18:39] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [10:48:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [13:48:39] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [14:18:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:18:40] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [16:48:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [18:18:39] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [18:48:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [19:08:47] [02WikiTideOrg/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/cache-3.3.3 [+0/-0/±1] 13https://github.com/WikiTideOrg/.github/commit/fab644ad81dd [19:08:49] [02WikiTideOrg/.github] 07dependabot[bot] 03fab644a - Bump actions/cache from 3.3.2 to 3.3.3 Bumps [actions/cache](https://github.com/actions/cache) from 3.3.2 to 3.3.3. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v3.3.2...v3.3.3) --- updated-dependencies: - dependency-name: actions/cache dependency-ty [19:08:50] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/actions/cache-3.3.3 - 13https://github.com/WikiTideOrg/.github [19:08:51] [02.github] 07dependabot[bot] opened pull request 03#1: Bump actions/cache from 3.3.2 to 3.3.3 - 13https://github.com/WikiTideOrg/.github/pull/1 [19:08:54] [02.github] 07dependabot[bot] labeled pull request 03#1: Bump actions/cache from 3.3.2 to 3.3.3 - 13https://github.com/WikiTideOrg/.github/pull/1 [19:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [19:18:35] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [21:11:22] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/6e27033591a8...ad60be87a991 [21:11:25] [02WikiTideOrg/mw-config] 07Universal-Omega 03ad60be8 - Update SMTP [21:12:16] WikiTideOrg/mw-config - Universal-Omega the build passed. [21:31:35] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [21:31:35] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [21:31:42] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:31:48] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [22:02:51] PROBLEM - swiftproxy1 SSH on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:01] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:01] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:12] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:13] PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:45] PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:56] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:08:32] RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set [22:08:33] RECOVERY - swiftproxy1 conntrack_table_size on swiftproxy1 is OK: OK: nf_conntrack is 0 % full [22:16:27] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset -0.009792864323 secs [22:18:22] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [22:18:24] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [22:18:57] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:19:02] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2628 bytes in 0.684 second response time [22:19:03] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2628 bytes in 0.295 second response time [22:19:11] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:19:21] PROBLEM - cp6 HTTPS on cp6 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2627 bytes in 0.603 second response time [22:20:20] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [22:20:54] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 3.583 second response time [22:20:58] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2952 bytes in 0.615 second response time [22:21:01] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2974 bytes in 1.350 second response time [22:21:07] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.158 second response time [22:21:20] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2995 bytes in 1.394 second response time [22:21:54] PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:21:54] PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:22:23] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [22:23:20] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:28:23] PROBLEM - swiftproxy1 Disk Space on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:31:25] RECOVERY - swiftproxy1 Disk Space on swiftproxy1 is OK: DISK OK - free space: / 23335MiB (87% inode=95%); [22:33:52] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [22:35:46] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [22:40:34] PROBLEM - swiftproxy1 Disk Space on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:42:13] RECOVERY - swiftproxy1 SSH on swiftproxy1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:42:17] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 0.336 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [22:42:31] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is WARNING: WARNING: Puppet last ran 1 hour ago [22:42:34] RECOVERY - swiftproxy1 Disk Space on swiftproxy1 is OK: DISK OK - free space: / 23333MiB (87% inode=95%); [22:42:52] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset -0.01802459359 secs [22:43:24] RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set [22:43:25] RECOVERY - swiftproxy1 conntrack_table_size on swiftproxy1 is OK: OK: nf_conntrack is 1 % full [22:44:26] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:12:12] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 0.45, 0.37, 7.64 [23:14:12] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.41, 0.36, 6.75