[00:12:04] PROBLEM - feathercare.net - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:24:39] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:52:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:50:38] PROBLEM - feathercare.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for feathercare.net could not be found [04:10:12] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/af69430c985c...2d6d499718c0 [04:10:14] [02ssl] 07WikiTideSSLBot 032d6d499 - Bot: Update SSL cert for rosettacode.org [04:25:18] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [04:54:16] RECOVERY - rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Mon 26 Aug 2024 03:10:07 AM GMT +0000. [04:54:49] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:12:10] RECOVERY - www.rosettacode.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rosettacode.org' will expire on Mon 26 Aug 2024 03:10:07 AM GMT +0000. [08:53:26] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:22:54] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:40:29] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864186 seconds old and 97655 bytes [10:06:27] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:11:22] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mystiverse.wiki All nameservers failed to answer the query. [10:23:51] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:30:39] PROBLEM - nijigen.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nijigen.wiki All nameservers failed to answer the query. [10:34:27] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [10:41:22] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mystiverse.wiki could not be found [10:53:16] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:00:40] PROBLEM - nijigen.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nijigen.wiki could not be found [14:31:40] [02mw-config] 07redbluegreenhat closed pull request 03#5578: Bing: increase crawl-delay - 13https://github.com/miraheze/mw-config/pull/5578 [14:31:42] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6cbcb139d501...1ea702007c14 [14:31:45] [02mw-config] 07RhinosF1 031ea7020 - Bing: increase crawl-delay (#5578) [14:32:34] !log [@mwtask181] starting deploy of {'config': True} to all [14:32:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:44] miraheze/mw-config - redbluegreenhat the build passed. [14:32:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [14:32:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:32] !log [@mwtask171] starting deploy of {'config': True} to all [14:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:37:41] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [14:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:54:06] !log [@test151] starting deploy of {'config': True} to test151 [14:54:07] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [14:54:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:54:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:41:22] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mystiverse.wiki All nameservers failed to answer the query. [15:54:02] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [16:11:22] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mystiverse.wiki could not be found [16:25:30] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:54:57] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [17:24:25] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:53:49] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [18:49:55] [02mw-config] 07songnguxyz opened pull request 03#5580: Adding new footer icons (StarCitizenTools style) to lhmnwiki and sister projects - 13https://github.com/miraheze/mw-config/pull/5580 [18:50:53] miraheze/mw-config - songnguxyz the build has errored. [18:52:34] [02mw-config] 07songnguxyz synchronize pull request 03#5580: Adding new footer icons (StarCitizenTools style) to lhmnwiki and sister projects - 13https://github.com/miraheze/mw-config/pull/5580 [18:53:24] miraheze/mw-config - songnguxyz the build has errored. [18:56:48] [02mw-config] 07songnguxyz synchronize pull request 03#5580: Adding new footer icons (StarCitizenTools style) to lhmnwiki and sister projects - 13https://github.com/miraheze/mw-config/pull/5580 [18:57:44] miraheze/mw-config - songnguxyz the build passed. [20:23:38] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [20:25:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [20:54:40] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [21:06:27] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:34:27] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [22:37:37] [02mw-config] 07clarfonthey commented on pull request 03#5580: Adding new footer icons (StarCitizenTools style) to lhmnwiki and sister projects - 13https://github.com/miraheze/mw-config/pull/5580#issuecomment-2136216034