[00:27:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:57:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:08:17] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[01:36:56] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[03:39:21] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[04:08:02] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[06:00:09] <botifico>	 [02mw-config] 07BlankEclair opened pull request 03#5590: T12143: Add "interwiki" to $wgRestrictionLevels for gengbaikewiki - 13https://github.com/miraheze/mw-config/pull/5590
[06:01:11] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[06:27:46] <botifico>	 [02MirahezeMagic] 07colleirose commented on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2171088151
[06:27:54] <botifico>	 [02MirahezeMagic] 07colleirose edited a comment on pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491#issuecomment-2171088151
[06:29:16] <botifico>	 [02MirahezeMagic] 07colleirose edited pull request 03#491: Add SEO tweaks - 13https://github.com/miraheze/MirahezeMagic/pull/491
[07:57:45] <MirahezeLSBot>	 !log [alex@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all
[07:57:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:57:56] <MirahezeLSBot>	 !log [alex@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 10s
[07:57:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:38:19] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[09:07:00] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[09:08:45] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[09:34:45] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[11:01:59] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[11:03:58] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001339018345 secs
[12:08:24] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[12:36:04] <icinga-miraheze>	 PROBLEM - wiki.cubestudios.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cubestudios.xyz' expires in 15 day(s) (Tue 02 Jul 2024 12:06:43 PM GMT +0000).
[12:36:15] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed6f253bb257...e89a741debab
[12:36:16] <botifico>	 [02ssl] 07WikiTideSSLBot 03e89a741 - Bot: Update SSL cert for wiki.cubestudios.xyz
[12:37:07] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[12:48:43] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[12:54:53] <icinga-miraheze>	 RECOVERY - wiki.sequoiafabrica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sequoiafabrica.org' will expire on Sat 14 Sep 2024 11:26:16 AM GMT +0000.
[12:57:52] <icinga-miraheze>	 RECOVERY - wiki.showtapeselection.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.showtapeselection.com' will expire on Sat 14 Sep 2024 11:28:31 AM GMT +0000.
[13:05:38] <icinga-miraheze>	 RECOVERY - wiki.cubestudios.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cubestudios.xyz' will expire on Sat 14 Sep 2024 11:36:09 AM GMT +0000.
[13:08:17] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[13:13:42] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures
[13:36:58] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[13:59:35] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[15:12:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[15:27:13] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000.
[15:36:16] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[15:41:03] <botifico>	 [02MirahezeMagic] 07joelemiliano opened pull request 03#492: translating to Español - 13https://github.com/miraheze/MirahezeMagic/pull/492
[15:41:08] <botifico>	 [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#492: translating to Español - 13https://github.com/miraheze/MirahezeMagic/pull/492#issuecomment-2171747670
[15:49:50] <botifico>	 [02MirahezeMagic] 07RhinosF1 commented on pull request 03#492: translating to Español - 13https://github.com/miraheze/MirahezeMagic/pull/492#issuecomment-2171749936
[15:51:12] <botifico>	 [02MirahezeMagic] 07RhinosF1 commented on pull request 03#492: translating to Español - 13https://github.com/miraheze/MirahezeMagic/pull/492#issuecomment-2171750330
[15:54:40] <botifico>	 [02MirahezeMagic] 07OAuthority closed pull request 03#492: translating to Español - 13https://github.com/miraheze/MirahezeMagic/pull/492
[16:16:52] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[16:46:06] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:06:09] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[19:02:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[20:27:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:02:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:22:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1