[00:00:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:53] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:03:21] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:06:29] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:06:42] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:06:52] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:09:03] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:09:32] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:11:03] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [00:11:18] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:11:31] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:01] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:14:25] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [00:15:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [00:15:19] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:18:27] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:18:40] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:19:02] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [00:19:18] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:20:27] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:21:20] RECOVERY - matomo1 Puppet on matomo1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [00:21:35] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:22:05] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:22:24] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:23:28] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:25:11] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:26:24] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:26:26] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:29:20] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:39:54] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/1268fc51b00e...cb0089bde932 [00:39:57] [02WikiTideOrg/dns] 07Universal-Omega 03cb0089b - Use ns2 [00:44:40] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [01:43:07] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/cb0089bde932...6e439db8f0ff [01:43:09] [02WikiTideOrg/dns] 07Universal-Omega 036e439db - Revert [02:57:50] PROBLEM - cp3 ferm_active on cp3 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [02:58:44] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [03:03:50] RECOVERY - cp3 ferm_active on cp3 is OK: OK ferm input default policy is set [03:13:26] PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:15:24] RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 3.411 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:17:27] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03remove-wikitide.net [+0/-1/±0] 13https://github.com/WikiTideOrg/dns/commit/7ac62a83467a [03:17:30] [02WikiTideOrg/dns] 07Universal-Omega 037ac62a8 - Remove wikitide.net Moved to miraheze/dns [03:17:33] [02dns] 07Universal-Omega created branch 03remove-wikitide.net - 13https://github.com/WikiTideOrg/dns [03:17:34] [02dns] 07Universal-Omega opened pull request 03#4: Remove wikitide.net - 13https://github.com/WikiTideOrg/dns/pull/4 [03:17:53] [02dns] 07Universal-Omega closed pull request 03#4: Remove wikitide.net - 13https://github.com/WikiTideOrg/dns/pull/4 [03:17:57] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/WikiTideOrg/dns/compare/6e439db8f0ff...ec382fc84ceb [03:17:59] [02WikiTideOrg/dns] 07Universal-Omega 03ec382fc - Remove wikitide.net Moved to miraheze/dns [03:18:01] [02WikiTideOrg/dns] 07Universal-Omega deleted branch 03remove-wikitide.net [03:18:02] [02dns] 07Universal-Omega deleted branch 03remove-wikitide.net - 13https://github.com/WikiTideOrg/dns [03:22:12] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.lophocmatngu.wiki All nameservers failed to answer the query. [03:23:24] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - alpha.sagan4.org All nameservers failed to answer the query. [03:23:40] PROBLEM - cp3 Auth DNS on cp3 is CRITICAL: Connection to DNS 51.75.170.66 was refused [03:24:28] PROBLEM - cp1 PowerDNS Recursor on cp1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:24:37] PROBLEM - mlrpgspeedruns.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [03:24:42] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 933 bytes in 0.101 second response time [03:24:55] PROBLEM - test1 PowerDNS Recursor on test1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:25:14] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: Connection to DNS 63.141.240.4 was refused [03:25:30] PROBLEM - mon1 PowerDNS Recursor on mon1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:25:30] PROBLEM - mail1 PowerDNS Recursor on mail1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:25:35] @cosmicalpha did you shut down ns*? [03:25:41] that brought everything down [03:25:42] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.colleimadcat.com All nameservers failed to answer the query. [03:25:46] PROBLEM - swiftac1 PowerDNS Recursor on swiftac1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:25:47] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2884 bytes in 0.086 second response time [03:25:51] PROBLEM - ldap1 LDAP on ldap1 is CRITICAL: Could not bind to the LDAP server [03:25:57] PROBLEM - prometheus1 PowerDNS Recursor on prometheus1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:06] oh geez, I didn't realize everything crashed if ns did [03:26:10] PROBLEM - phorge1 PowerDNS Recursor on phorge1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:22] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:34] PROBLEM - swiftobject1 PowerDNS Recursor on swiftobject1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:39] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03revert-4-remove-wikitide.net [+1/-0/±0] 13https://github.com/WikiTideOrg/dns/commit/ccf84e203521 [03:26:41] PROBLEM - bast1 PowerDNS Recursor on bast1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:41] [02WikiTideOrg/dns] 07Universal-Omega 03ccf84e2 - Revert "Remove wikitide.net" [03:26:42] [02dns] 07Universal-Omega created branch 03revert-4-remove-wikitide.net - 13https://github.com/WikiTideOrg/dns [03:26:43] [02dns] 07Universal-Omega opened pull request 03#5: Revert "Remove wikitide.net" - 13https://github.com/WikiTideOrg/dns/pull/5 [03:26:43] PROBLEM - mw2 PowerDNS Recursor on mw2 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:54] [02dns] 07Universal-Omega closed pull request 03#5: Revert "Remove wikitide.net" - 13https://github.com/WikiTideOrg/dns/pull/5 [03:26:56] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiTideOrg/dns/compare/ec382fc84ceb...390e547097df [03:26:58] PROBLEM - mw1 PowerDNS Recursor on mw1 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:26:59] [02WikiTideOrg/dns] 07Universal-Omega 03390e547 - Revert "Remove wikitide.net" (#5) [03:27:00] [02WikiTideOrg/dns] 07Universal-Omega deleted branch 03revert-4-remove-wikitide.net [03:27:02] [02dns] 07Universal-Omega deleted branch 03revert-4-remove-wikitide.net - 13https://github.com/WikiTideOrg/dns [03:27:13] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 933 bytes in 0.125 second response time [03:27:45] RECOVERY - cp3 Auth DNS on cp3 is OK: DNS OK: 5.255 seconds response time. wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66 [03:27:46] RECOVERY - swiftac1 PowerDNS Recursor on swiftac1 is OK: DNS OK: 0.496 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:27:47] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.164 second response time [03:27:51] RECOVERY - ldap1 LDAP on ldap1 is OK: LDAP OK - 0.216 seconds response time [03:27:52] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: Domain 'wikitide.org' was not found by the server [03:27:58] RECOVERY - prometheus1 PowerDNS Recursor on prometheus1 is OK: DNS OK: 0.381 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:10] RECOVERY - phorge1 PowerDNS Recursor on phorge1 is OK: DNS OK: 0.378 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:23] RECOVERY - cp5 PowerDNS Recursor on cp5 is OK: DNS OK: 0.924 seconds response time. wikitide.org returns 15.235.167.159,2402:1f00:8000:800::1b3d [03:28:29] RECOVERY - cp1 PowerDNS Recursor on cp1 is OK: DNS OK: 6.801 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:34] RECOVERY - swiftobject1 PowerDNS Recursor on swiftobject1 is OK: DNS OK: 0.424 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:40] RECOVERY - bast1 PowerDNS Recursor on bast1 is OK: DNS OK: 0.363 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:42] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.157 second response time [03:28:42] RECOVERY - mw2 PowerDNS Recursor on mw2 is OK: DNS OK: 0.238 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:54] RECOVERY - test1 PowerDNS Recursor on test1 is OK: DNS OK: 0.462 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:28:57] RECOVERY - mw1 PowerDNS Recursor on mw1 is OK: DNS OK: 0.262 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:29:13] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.152 second response time [03:29:30] RECOVERY - mon1 PowerDNS Recursor on mon1 is OK: DNS OK: 0.038 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:29:31] RECOVERY - mail1 PowerDNS Recursor on mail1 is OK: DNS OK: 0.479 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [03:29:52] RECOVERY - cp6 PowerDNS Recursor on cp6 is OK: DNS OK: 1.813 second response time. wikitide.org returns 139.99.236.151,2402:1f00:8100:400::1d23 [03:30:43] PROBLEM - wiki.burning.art - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.burning.art could not be found [03:30:54] I did not, I removed wikitide.net, before it was fully updated to be on MH apparently, the internal networking didn't know it changed but my browser did [03:31:14] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.038 seconds response time. wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66 [03:32:13] That was a bit odd lol but back up now as I reverted removing wikitide.net from our dns here. [03:32:44] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [03:51:22] PROBLEM - data.lophocmatngu.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['lia.ns.cloudflare.com.', 'gerald.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:52:18] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [03:53:32] RECOVERY - mlrpgspeedruns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mlrpgspeedruns.com' will expire on Mon 04 Mar 2024 02:14:22 PM GMT +0000. [03:55:00] PROBLEM - wiki.colleimadcat.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns29.hichina.com.', 'dns30.hichina.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [05:04:18] PROBLEM - wikitide.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikitide.net could not be found [08:30:52] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:31:40] PROBLEM - swiftproxy1 SSH on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:26] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [08:33:41] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:34:51] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 1.464 second response time [08:35:26] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [08:35:40] PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:01] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:26] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:31] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:36:31] PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:38:33] RECOVERY - swiftproxy1 conntrack_table_size on swiftproxy1 is OK: OK: nf_conntrack is 0 % full [08:38:33] RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set [08:38:35] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 0.435 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [08:38:36] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 39 minutes ago with 0 failures [08:38:39] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset 0.005350738764 secs [08:40:07] RECOVERY - swiftproxy1 SSH on swiftproxy1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [08:57:34] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 0.11, 1.19, 7.46 [08:59:28] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.41, 0.97, 6.68 [09:25:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4faad1aae736...5057a61f622f [09:25:17] [02WikiTideOrg/puppet] 07Universal-Omega 035057a61 - phorge: fix SMTP [11:02:54] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [11:32:11] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [17:37:46] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:40:15] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:41:47] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:41:53] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:41:57] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:44:46] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:45:32] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:05:46] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:33:30] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:37:53] PROBLEM - graylog1 Puppet on graylog1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:07:46] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:08:38] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:35:46] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:36:37] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:46:09] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:46:38] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:47] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:56] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:50:31] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:51:17] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:52:05] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:53:01] PROBLEM - bots1 Puppet on bots1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:53:19] PROBLEM - matomo1 Puppet on matomo1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:53:53] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:01] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:29] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:57:17] PROBLEM - db1 Puppet on db1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:57:35] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:59:35] PROBLEM - cp1 Puppet on cp1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:01:46] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:46] PROBLEM - swiftac1 Puppet on swiftac1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:07:46] PROBLEM - mem1 Puppet on mem1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:08:35] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:12:55] PROBLEM - www.wikitide.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.miraheze.org.', 'ns2.miraheze.org.'], 'CNAME': None} [20:13:30] PROBLEM - mlrpgspeedruns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mlrpgspeedruns.com All nameservers failed to answer the query. [20:16:50] PROBLEM - cp3 GDNSD Datacenters on cp3 is UNKNOWN: NRPE: Unable to read output [20:17:05] PROBLEM - sdiy.info - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.miraheze.org.', 'ns2.miraheze.org.'], 'CNAME': None} [20:17:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [20:18:27] !log stop gdnsd on ns* [20:18:34] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:18:54] PROBLEM - test1 PowerDNS Recursor on test1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:18:59] PROBLEM - mw1 PowerDNS Recursor on mw1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:05] PROBLEM - services1 PowerDNS Recursor on services1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:11] PROBLEM - mail1 PowerDNS Recursor on mail1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:13] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:16] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:23] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: No response from DNS 63.141.240.4 [20:19:29] PROBLEM - matomo1 PowerDNS Recursor on matomo1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:30] PROBLEM - mon1 PowerDNS Recursor on mon1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:40] PROBLEM - cp3 Auth DNS on cp3 is CRITICAL: No response from DNS 51.75.170.66 [20:19:49] PROBLEM - bots1 PowerDNS Recursor on bots1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:19:52] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:20:29] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.miraheze.org.', 'ns2.miraheze.org.'], 'CNAME': None} [20:20:44] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:20:53] PROBLEM - cp4 PowerDNS Recursor on cp4 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:21:00] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5057a61f622f...563e3c8e3b7e [20:21:01] [02WikiTideOrg/puppet] 07Universal-Omega 03563e3c8 - check_reverse_dns: use miraheze nameservers [20:21:24] PROBLEM - jobchron1 PowerDNS Recursor on jobchron1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:21:47] PROBLEM - swiftac1 PowerDNS Recursor on swiftac1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:21:49] RECOVERY - bots1 PowerDNS Recursor on bots1 is OK: DNS OK: 0.201 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:21:52] RECOVERY - cp6 PowerDNS Recursor on cp6 is OK: DNS OK: 1.146 second response time. wikitide.org returns 139.99.236.151,2402:1f00:8100:400::1d23 [20:21:57] PROBLEM - prometheus1 PowerDNS Recursor on prometheus1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:22:12] PROBLEM - phorge1 PowerDNS Recursor on phorge1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:22:17] WikiTideOrg/puppet - Universal-Omega the build passed. [20:22:33] PROBLEM - swiftobject1 PowerDNS Recursor on swiftobject1 is CRITICAL: Domain 'wikitide.org' was not found by the server [20:22:50] PROBLEM - cp3 GDNSD Datacenters on cp3 is CRITICAL: CRITICAL - 3 datacenters are down: 51.79.55.151/cpweb, 2607:5300:205:200::2aa8/cpweb, 63.141.240.6/cpweb [20:22:53] RECOVERY - cp4 PowerDNS Recursor on cp4 is OK: DNS OK: 0.282 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [20:22:54] RECOVERY - test1 PowerDNS Recursor on test1 is OK: DNS OK: 0.349 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:22:57] RECOVERY - mw1 PowerDNS Recursor on mw1 is OK: DNS OK: 0.183 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:22:59] PROBLEM - www.rosettacode.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.miraheze.org.', 'ns2.miraheze.org.'], 'CNAME': None} [20:23:06] RECOVERY - services1 PowerDNS Recursor on services1 is OK: DNS OK: 0.416 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:12] RECOVERY - mail1 PowerDNS Recursor on mail1 is OK: DNS OK: 0.503 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:13] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 0.181 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.79.55.151/cpweb, 2607:5300:205:200::2aa8/cpweb, 63.141.240.6/cpweb [20:23:20] RECOVERY - cp5 PowerDNS Recursor on cp5 is OK: DNS OK: 0.950 seconds response time. wikitide.org returns 15.235.167.159,2402:1f00:8000:800::1b3d [20:23:22] RECOVERY - jobchron1 PowerDNS Recursor on jobchron1 is OK: DNS OK: 0.178 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:26] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.034 seconds response time. wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:26] RECOVERY - matomo1 PowerDNS Recursor on matomo1 is OK: DNS OK: 0.183 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:30] RECOVERY - mon1 PowerDNS Recursor on mon1 is OK: DNS OK: 0.035 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:40] RECOVERY - cp3 Auth DNS on cp3 is OK: DNS OK: 0.223 seconds response time. wikitide.net returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:46] RECOVERY - swiftac1 PowerDNS Recursor on swiftac1 is OK: DNS OK: 0.176 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:23:49] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.miraheze.org.', 'ns2.miraheze.org.'], 'CNAME': None} [20:23:58] RECOVERY - prometheus1 PowerDNS Recursor on prometheus1 is OK: DNS OK: 0.142 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:24:09] RECOVERY - phorge1 PowerDNS Recursor on phorge1 is OK: DNS OK: 0.028 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:24:33] RECOVERY - swiftobject1 PowerDNS Recursor on swiftobject1 is OK: DNS OK: 0.285 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [20:28:26] PROBLEM - wikitide.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.miraheze.org.', 'ns2.miraheze.org.'], 'CNAME': None} [20:29:53] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/563e3c8e3b7e...6bc524bc560f [20:29:56] [02WikiTideOrg/puppet] 07Universal-Omega 036bc524b - Update recursor [20:31:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6bc524bc560f...c2de5c307731 [20:31:40] [02WikiTideOrg/puppet] 07Universal-Omega 03c2de5c3 - Update resolver [20:38:31] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:39:46] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:40:03] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:41:49] RECOVERY - www.wikitide.org - reverse DNS on sslhost is OK: SSL OK - www.wikitide.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [20:41:55] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:41:57] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:42:47] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:43:32] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:44:09] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:44:40] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:45:36] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [20:45:37] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:45:40] RECOVERY - sdiy.info - reverse DNS on sslhost is OK: SSL OK - sdiy.info reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [20:45:46] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:45:53] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:45:53] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [20:45:55] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [20:46:04] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures [20:46:28] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:46:35] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [20:46:44] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:47:01] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:01] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:17] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:19] RECOVERY - matomo1 Puppet on matomo1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:27] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:27] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:37] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:47:46] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:47:46] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:48:11] RECOVERY - nexttide.org - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [20:48:50] PROBLEM - cp3 GDNSD Datacenters on cp3 is UNKNOWN: NRPE: Unable to read output [20:49:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [20:49:29] RECOVERY - rosettacode.org - reverse DNS on sslhost is OK: SSL OK - rosettacode.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [20:51:14] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: No response from DNS 63.141.240.4 [20:51:40] PROBLEM - cp3 Auth DNS on cp3 is CRITICAL: No response from DNS 51.75.170.66 [20:52:23] RECOVERY - www.rosettacode.org - reverse DNS on sslhost is OK: SSL OK - www.rosettacode.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [20:53:39] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [20:57:52] RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp3.wikitide.net - NS RECORDS OK [20:59:50] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03cp3-no-dns [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/644cd6d2d35c [20:59:51] [02WikiTideOrg/puppet] 07Universal-Omega 03644cd6d - cp3: remove dns [20:59:54] [02puppet] 07Universal-Omega created branch 03cp3-no-dns - 13https://github.com/WikiTideOrg/puppet [20:59:56] [02puppet] 07Universal-Omega opened pull request 03#47: cp3: remove dns - 13https://github.com/WikiTideOrg/puppet/pull/47 [21:00:21] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03cp3-no-dns [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/644cd6d2d35c...fe80d479ee7b [21:00:23] [02WikiTideOrg/puppet] 07Universal-Omega 03fe80d47 - Remove [21:00:24] [02puppet] 07Universal-Omega synchronize pull request 03#47: cp3: remove dns - 13https://github.com/WikiTideOrg/puppet/pull/47 [21:03:14] [02puppet] 07Universal-Omega closed pull request 03#47: cp3: remove dns - 13https://github.com/WikiTideOrg/puppet/pull/47 [21:03:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/c2de5c307731...b0a13ecdf8f3 [21:03:18] [02WikiTideOrg/puppet] 07Universal-Omega 03b0a13ec - cp3: remove dns [21:03:20] RECOVERY - newcascadia.net - LetsEncrypt on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [21:03:21] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03cp3-no-dns [21:03:22] [02puppet] 07Universal-Omega deleted branch 03cp3-no-dns - 13https://github.com/WikiTideOrg/puppet [21:04:08] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:20:32] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03remove-ns1 [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/1ab744e94d22 [21:20:33] [02WikiTideOrg/puppet] 07Universal-Omega 031ab744e - Remove ns1 [21:20:35] [02puppet] 07Universal-Omega created branch 03remove-ns1 - 13https://github.com/WikiTideOrg/puppet [21:20:37] [02puppet] 07Universal-Omega opened pull request 03#48: Remove ns1 - 13https://github.com/WikiTideOrg/puppet/pull/48 [21:21:02] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03remove-ns1 [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/1ab744e94d22...83031000e797 [21:21:05] [02WikiTideOrg/puppet] 07Universal-Omega 038303100 - Delete hieradata/hosts/ns1.yaml [21:21:08] [02puppet] 07Universal-Omega synchronize pull request 03#48: Remove ns1 - 13https://github.com/WikiTideOrg/puppet/pull/48 [21:21:57] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b0a13ecdf8f3...cf536111cc3c [21:21:58] [02WikiTideOrg/puppet] 07Universal-Omega 03cf53611 - Remove mail1 [21:22:50] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/cf536111cc3c...d69d55d64896 [21:22:51] [02WikiTideOrg/puppet] 07Universal-Omega 03d69d55d - Remove mail1 [21:25:59] [02puppet] 07Universal-Omega closed pull request 03#48: Remove ns1 - 13https://github.com/WikiTideOrg/puppet/pull/48 [21:26:01] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/WikiTideOrg/puppet/compare/d69d55d64896...f4a160066550 [21:26:04] [02WikiTideOrg/puppet] 07Universal-Omega 03f4a1600 - Remove ns1 [21:26:05] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03remove-ns1 [21:26:08] [02puppet] 07Universal-Omega deleted branch 03remove-ns1 - 13https://github.com/WikiTideOrg/puppet [21:27:00] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/f4a160066550...991ef13d8b75 [21:27:01] [02WikiTideOrg/puppet] 07Universal-Omega 03991ef13 - Remove [21:28:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: connect to address 63.141.240.4 port 5666: No route to hostconnect to host 63.141.240.4 port 5666: No route to host [21:31:36] !log remove ns1 and mail1 from puppetserver and deactivate nodes, then shutdown [21:31:43] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:33:12] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/390e547097df...2673b9cb28e1 [21:33:14] [02WikiTideOrg/dns] 07Universal-Omega 032673b9c - ARCHIVE [21:37:37] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:39:37] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.170 second response time [21:50:08] PROBLEM - cp6 Nginx Backend for mail1 on cp6 is CRITICAL: connect to address localhost and port 8202: Connection refused [21:53:47] RECOVERY - hsck.lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Fri 12 Apr 2024 11:08:03 PM GMT +0000. [21:53:59] PROBLEM - cp3 Nginx Backend for mail1 on cp3 is CRITICAL: connect to address localhost and port 8202: Connection refused [21:54:51] PROBLEM - cp5 Nginx Backend for mail1 on cp5 is CRITICAL: connect to address localhost and port 8202: Connection refused [21:55:57] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 05 Apr 2024 11:35:24 PM GMT +0000. [21:55:59] RECOVERY - www.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Fri 05 Apr 2024 11:07:53 PM GMT +0000. [21:57:20] RECOVERY - lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Fri 12 Apr 2024 11:08:03 PM GMT +0000. [22:00:26] PROBLEM - cp1 Nginx Backend for mail1 on cp1 is CRITICAL: connect to address localhost and port 8202: Connection refused [22:00:33] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Fri 05 Apr 2024 11:07:53 PM GMT +0000. [22:01:15] RECOVERY - www.lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 05 Apr 2024 11:40:36 PM GMT +0000. [22:02:16] RECOVERY - distrowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'distrowiki.net' will expire on Fri 05 Apr 2024 11:46:50 PM GMT +0000. [22:03:55] RECOVERY - alpha.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Fri 05 Apr 2024 11:29:24 PM GMT +0000. [22:04:00] RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Fri 05 Apr 2024 11:35:24 PM GMT +0000. [22:06:56] RECOVERY - valorantjp.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valorantjp.wiki' will expire on Fri 05 Apr 2024 11:17:32 PM GMT +0000. [22:07:11] RECOVERY - wiki.colleimadcat.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.colleimadcat.com' will expire on Fri 05 Apr 2024 11:23:01 PM GMT +0000. [22:07:38] RECOVERY - meta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Fri 05 Apr 2024 11:29:24 PM GMT +0000. [22:08:30] RECOVERY - www.greatamerica.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'greatamerica.wiki' will expire on Fri 05 Apr 2024 11:24:34 PM GMT +0000. [22:14:48] RECOVERY - beta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Fri 05 Apr 2024 11:51:59 PM GMT +0000. [22:16:05] RECOVERY - mason.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Fri 05 Apr 2024 11:51:59 PM GMT +0000. [22:17:54] RECOVERY - thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thegreatwar.uk' will expire on Fri 05 Apr 2024 11:59:18 PM GMT +0000. [22:18:34] RECOVERY - greatamerica.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'greatamerica.wiki' will expire on Fri 05 Apr 2024 11:24:34 PM GMT +0000.