[00:06:58] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:53:39] !log [@test151] starting deploy of {'config': True} to test151 [00:53:40] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:53:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:38:05] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:06:22] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:49:23] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Thu 09 May 2024 02:41:46 AM GMT +0000). [02:54:15] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mwcosmos.com' expires in 15 day(s) (Thu 09 May 2024 02:41:46 AM GMT +0000). [02:54:42] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2d04853da3cd...13c1ed23b154 [02:54:45] [02miraheze/ssl] 07WikiTideSSLBot 0313c1ed2 - Bot: Update SSL cert for mwcosmos.com [03:09:03] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:23:58] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Jul 2024 01:54:35 AM GMT +0000. [03:37:22] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:37:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:48:12] RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Jul 2024 01:54:35 AM GMT +0000. [04:25:24] PROBLEM - db162 MariaDB on db162 is UNKNOWN: check_mysql: Invalid hostname/address - db162.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group] [04:27:22] PROBLEM - db162 MariaDB on db162 is CRITICAL: Access denied for user 'icinga'@'2602:294:0:b12::110' (using password: YES) [04:37:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:32:13] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.43 ms [05:34:13] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.79 ms [06:17:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:22:50] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.15 ms [06:24:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.82 ms [06:58:13] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [07:02:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:09:34] !log [@mwtask171] starting deploy of {'config': True} to all [07:09:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:09:45] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 11s [07:09:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:07] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:15:22] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (8 critical updates). [07:17:18] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:17:21] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:18:13] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 9 packages available for upgrade (8 critical updates). [07:18:55] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:25:30] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:26:07] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:26:32] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:26:47] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:27:05] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (8 critical updates). [07:27:28] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [07:34:19] !log [@mwtask181] starting deploy of {'config': True} to all [07:34:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:31] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 12s [07:34:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:10:21] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:10:50] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:38:50] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [08:45:18] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:09:39] !log [@mwtask171] starting deploy of {'config': True} to all [09:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:09:50] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [09:09:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:02] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:44:53] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:11:39] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [10:37:17] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.82 ms [10:41:25] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:43:20] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.90 ms [11:03:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.18 ms [11:05:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.30 ms [11:26:23] PROBLEM - polcompball.wikitide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wikitide.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:26:34] PROBLEM - wikitide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wikitide.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:31:50] PROBLEM - cheeseepedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cheeseepedia.org' expires in 15 day(s) (Thu 09 May 2024 11:09:50 AM GMT +0000). [11:32:02] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13c1ed23b154...1fd77e4c5276 [11:32:05] [02miraheze/ssl] 07WikiTideSSLBot 031fd77e4 - Bot: Update SSL cert for cheeseepedia.org [11:37:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:55:07] RECOVERY - polcompball.wikitide.org - reverse DNS on sslhost is OK: SSL OK - polcompball.wikitide.org reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [11:55:31] RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [12:01:43] RECOVERY - cheeseepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cheeseepedia.org' will expire on Mon 22 Jul 2024 10:31:56 AM GMT +0000. [12:07:51] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:19:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.18 ms [12:23:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.56 ms [12:36:19] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [12:54:04] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 165.86 ms [12:56:04] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 164.04 ms [13:16:33] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:50:27] !log [alex@mw151] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:50:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:55] !log [alex@mw152] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:51:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:51:30] RECOVERY - mw152 APT on mw152 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:51:58] RECOVERY - mw151 APT on mw151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:52:29] !log [alex@mw161] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:52:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:53:24] !log [alex@mw162] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:53:30] RECOVERY - mw161 APT on mw161 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:53:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:30] !log [alex@mw171] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:54:33] RECOVERY - mw162 APT on mw162 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:54:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:53] RECOVERY - mw171 APT on mw171 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:55:00] !log [alex@mw172] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:55:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:40] !log [alex@mw181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:55:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:18] !log [alex@mw182] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:47] RECOVERY - mw181 APT on mw181 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:56:52] RECOVERY - mw172 APT on mw172 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:57:18] RECOVERY - mw182 APT on mw182 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [13:58:07] !log [alex@mwtask171] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:58:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:24] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [13:58:43] !log [alex@mwtask181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [13:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:59:22] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:31:10] !log [alex@test151] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all [14:31:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:31:41] RECOVERY - test151 APT on test151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [14:33:34] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:36:42] !log rebooted test151 [14:36:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:26] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:04:01] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/77e75c45b62e...9976594b5641 [15:04:03] [02miraheze/dns] 07Universal-Omega 039976594 - Remove dccomicswiki.com zone [15:06:21] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9976594b5641...2718e427faa8 [15:06:24] [02miraheze/dns] 07Universal-Omega 032718e42 - Add accounts.wikitide.net [15:11:16] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:30:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03internal [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5ea4795d78a1 [15:30:37] [02miraheze/mw-config] 07Universal-Omega 035ea4795 - Add internal realm [15:30:38] [02mw-config] 07Universal-Omega created branch 03internal - 13https://github.com/miraheze/mw-config [15:30:39] [02mw-config] 07Universal-Omega opened pull request 03#5543: Add internal realm - 13https://github.com/miraheze/mw-config/pull/5543 [15:31:29] miraheze/mw-config - Universal-Omega the build has errored. [15:35:19] [02mw-config] 07Universal-Omega synchronize pull request 03#5543: Add internal realm - 13https://github.com/miraheze/mw-config/pull/5543 [15:35:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03internal [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5ea4795d78a1...bec462d34f95 [15:35:23] [02miraheze/mw-config] 07Universal-Omega 03bec462d - Fix [15:36:08] miraheze/mw-config - Universal-Omega the build passed. [15:37:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:45:43] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:00:58] !log rebooted mwtask171 [16:01:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:54] !log rebooted mwtask181 [16:06:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:17] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:13:56] !log rebooted mw151 [16:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:33] !log rebooted mw152 [16:16:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:11] !log rebooted mw161 [16:19:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:24] !log rebooted mw162 [16:26:18] !log (18:22) rebooted mw161 [16:26:27] !log (18:22) rebooted mw162 [16:26:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:28:55] !log rebooted mw171 [16:29:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:29] !log rebooted mw172 [16:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:34:12] !log rebooted mw181 [16:34:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:59] !log rebooted mw182 [16:36:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:51] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [17:06:40] PROBLEM - ping6 on cp27 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 136.83 ms [17:08:38] RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 137.10 ms [17:09:32] !log [@mwtask171] starting deploy of {'config': True} to all [17:09:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:43] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [17:09:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:40] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.04 ms [17:18:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.70 ms [17:20:48] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [17:53:19] PROBLEM - lophocmatngu.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lophocmatngu.wiki' expires in 15 day(s) (Thu 09 May 2024 05:32:10 PM GMT +0000). [17:53:28] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1fd77e4c5276...39dd15651d3e [17:53:31] [02miraheze/ssl] 07WikiTideSSLBot 0339dd156 - Bot: Update SSL cert for lophocmatngu.wiki [18:01:36] PROBLEM - sdiy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sdiy.info' expires in 15 day(s) (Thu 09 May 2024 05:43:19 PM GMT +0000). [18:01:43] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/39dd15651d3e...7f815c290c9f [18:01:46] [02miraheze/ssl] 07WikiTideSSLBot 037f815c2 - Bot: Update SSL cert for sdiy.info [18:03:28] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:10:33] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:22:57] RECOVERY - lophocmatngu.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lophocmatngu.wiki' will expire on Mon 22 Jul 2024 04:53:21 PM GMT +0000. [18:26:40] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.29 ms [18:28:40] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.45 ms [18:30:54] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Mon 22 Jul 2024 05:01:38 PM GMT +0000. [18:36:50] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:41:16] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:51:28] PROBLEM - tfuk.app - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [18:51:39] PROBLEM - 321nails.crpteam.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 321nails.crpteam.club could not be found [18:52:47] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [18:53:20] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ao90.pinho.org All nameservers failed to answer the query. [18:54:19] PROBLEM - cloud.neptune.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - cloud.neptune.wiki All nameservers failed to answer the query. [18:54:21] PROBLEM - burnout.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - burnout.wiki All nameservers failed to answer the query. [18:54:50] PROBLEM - schizoidnightmares.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - schizoidnightmares.wiki All nameservers failed to answer the query. [18:54:56] PROBLEM - touhouplace.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - touhouplace.wiki All nameservers failed to answer the query. [18:55:05] PROBLEM - monarchists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - monarchists.wiki All nameservers failed to answer the query. [18:55:07] PROBLEM - tech.exputra.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tech.exputra.com All nameservers failed to answer the query. [18:55:12] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.astralprojections.org All nameservers failed to answer the query. [18:55:19] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query. [18:55:32] PROBLEM - pandorastale.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pandorastale.wiki All nameservers failed to answer the query. [18:56:04] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query. [18:56:08] PROBLEM - wiki.blackorchestra.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.blackorchestra.net All nameservers failed to answer the query. [18:56:10] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mobilityengineer.com All nameservers failed to answer the query. [18:56:12] PROBLEM - wiki.cogsci.live - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.cogsci.live All nameservers failed to answer the query. [18:56:16] PROBLEM - project-patterns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - project-patterns.com All nameservers failed to answer the query. [18:56:28] PROBLEM - small.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - small.polandballwiki.com All nameservers failed to answer the query. [18:56:36] PROBLEM - evade.robtospro.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - evade.robtospro.com All nameservers failed to answer the query. [18:56:46] PROBLEM - grayravens.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - grayravens.com All nameservers failed to answer the query. [18:56:53] PROBLEM - fanon.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fanon.polandballwiki.com All nameservers failed to answer the query. [18:56:58] PROBLEM - www.langrisser.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.langrisser.wiki All nameservers failed to answer the query. [18:56:58] PROBLEM - aman.awiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - aman.awiki.org All nameservers failed to answer the query. [18:57:04] PROBLEM - wiki.sequoiafabrica.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sequoiafabrica.org All nameservers failed to answer the query. [18:57:05] PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.kirbygang.com All nameservers failed to answer the query. [18:57:05] PROBLEM - www.documentopublico.com.br - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.documentopublico.com.br All nameservers failed to answer the query. [18:57:23] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ssij.se All nameservers failed to answer the query. [18:57:37] PROBLEM - wiki.drycleaninglobby.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.drycleaninglobby.net All nameservers failed to answer the query. [18:57:39] PROBLEM - freedomplanetwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - freedomplanetwiki.com All nameservers failed to answer the query. [18:57:39] PROBLEM - pwsc.polandballwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pwsc.polandballwiki.com All nameservers failed to answer the query. [18:57:57] PROBLEM - webkinzguide.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - webkinzguide.com All nameservers failed to answer the query. [18:57:59] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - storytime.jdstroy.cf All nameservers failed to answer the query. [18:58:07] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.myehs.eu All nameservers failed to answer the query. [18:58:15] PROBLEM - wiki.simorgh.me - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.simorgh.me All nameservers failed to answer the query. [18:58:19] PROBLEM - wiki.openhatch.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.openhatch.org All nameservers failed to answer the query. [18:58:25] PROBLEM - sarovia.graalmilitary.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.408 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [18:58:40] PROBLEM - wiki.make717.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.make717.org All nameservers failed to answer the query. [19:00:19] PROBLEM - knifepointhorror.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - knifepointhorror.wiki All nameservers failed to answer the query. [19:01:04] PROBLEM - kafka181 NTP time on kafka181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o