[00:21:46] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 11.73, 10.53, 4.76
[00:23:46] <icinga-miraheze>	 RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 4.18, 7.76, 4.43
[00:27:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:57:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:23:19] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for private.yahyabd.xyz could not be found
[01:25:37] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[01:36:01] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yahyabd.xyz could not be found
[01:47:17] <icinga-miraheze>	 PROBLEM - db151 Backups SQL on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[01:49:12] <icinga-miraheze>	 PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 922718 seconds old and 124661 bytes
[05:26:42] <icinga-miraheze>	 PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: CRITICAL - Plugin timed out while executing system call
[05:27:48] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 85.06, 44.72, 18.79
[05:28:47] <icinga-miraheze>	 PROBLEM - db151 Puppet on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:30:46] <icinga-miraheze>	 RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures
[05:30:53] <icinga-miraheze>	 RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.058 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206
[05:43:46] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.23, 3.92, 11.27
[05:45:46] <icinga-miraheze>	 RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.30, 2.72, 9.93
[06:59:22] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[07:29:29] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[09:30:50] <botifico>	 [02mw-config] 07BlankEclair opened pull request 03#5572: Exclude www.deviantart.com and jocelynsamara.deviantart.com from RottenLinks testing on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5572
[09:30:55] <botifico>	 [02mw-config] 07coderabbitai[bot] commented on pull request 03#5572: Exclude www.deviantart.com and jocelynsamara.deviantart.com from RottenLinks testing on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5572#issuecomment-2126654578
[09:33:01] <botifico>	 [02mw-config] 07BlankEclair edited pull request 03#5572: Exclude www.deviantart.com and jocelynsamara.deviantart.com from RottenLinks testing on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5572
[10:23:47] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[10:53:34] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[12:15:17] <botifico>	 [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7d62a4c3098e...6917022422b7
[12:15:19] <botifico>	 [02ImportDump] 07translatewiki 036917022 - Localisation updates from https://translatewiki.net.
[12:15:21] <botifico>	 [02RequestSSL] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/d3cd494729f7...3c711da73bcb
[12:15:22] <botifico>	 [02RequestSSL] 07translatewiki 033c711da - Localisation updates from https://translatewiki.net.
[12:15:23] <botifico>	 [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±13] 13https://github.com/miraheze/MirahezeMagic/compare/a9eaf52a4ff0...f6afacc445f2
[12:15:26] <botifico>	 [02MirahezeMagic] 07translatewiki 03f6afacc - Localisation updates from https://translatewiki.net.
[12:15:28] <botifico>	 [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/99ebb9e132cd...ec4b68528b6c
[12:15:29] <botifico>	 [02DataDump] 07translatewiki 03ec4b685 - Localisation updates from https://translatewiki.net.
[12:20:04] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:20:09] <miraheze-github_>	 miraheze/DataDump - translatewiki the build passed.
[12:23:54] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:23:55] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:05] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:24:06] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:06] <miraheze-github>	 miraheze/RequestSSL - translatewiki the build passed.
[12:24:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:18] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[12:24:19] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:29] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:33:06] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:33:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:16] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s
[12:33:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:27] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:33:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:37] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[12:33:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:23] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:37:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:32] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:37:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:43] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:37:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:51] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:37:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:23:00] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.16 ms
[13:25:01] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 139.06 ms
[13:39:14] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 142.25 ms
[13:41:15] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 141.18 ms
[14:32:09] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.56 ms
[14:34:10] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 142.31 ms
[15:09:45] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 141.26 ms
[15:11:47] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 142.88 ms
[15:33:44] <MirahezeLSBot>	 !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=tallywiki /srv/mediawiki/1.41/vendor/mwstake/mediawiki-component-commonwebapis/sql/mws_category_index.sql (END - exit=0)
[15:33:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:49:30] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 141.33 ms
[15:53:34] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 140.77 ms
[16:00:49] <miraheze-github>	 miraheze/mw-config - BlankEclair the build passed.
[16:01:10] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5572: Exclude www.deviantart.com and jocelynsamara.deviantart.com from RottenLinks testing on rainversewiki - 13https://github.com/miraheze/mw-config/pull/5572
[16:01:12] <botifico>	 [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7e43d220815a...4a94d06400d7
[16:01:14] <botifico>	 [02mw-config] 07BlankEclair 034a94d06 - Exclude www.deviantart.com and jocelynsamara.deviantart.com from RottenLinks testing on rainversewiki (#5572)
[16:02:04] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[16:02:44] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[16:02:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:02:54] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s
[16:02:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:03:06] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5565: Merge duplicate entries together - 13https://github.com/miraheze/mw-config/pull/5565
[16:03:08] <botifico>	 [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4a94d06400d7...2c850c57b2f6
[16:03:10] <botifico>	 [02mw-config] 07colleirose 032c850c5 - Merge duplicate entries together (#5565)
[16:03:59] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[16:05:17] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5566: Changes to how special pages are treated - 13https://github.com/miraheze/mw-config/pull/5566
[16:05:18] <botifico>	 [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2c850c57b2f6...a4040a4737bc
[16:05:19] <botifico>	 [02mw-config] 07colleirose 03a4040a4 - Changes to how special pages are treated (#5566)
[16:06:20] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[16:07:24] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[16:07:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:07:32] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[16:07:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:08:29] <botifico>	 [02mw-config] 07Universal-Omega reviewed pull request 03#5569 commit - 13https://github.com/miraheze/mw-config/pull/5569#discussion_r1611964432
[16:08:31] <botifico>	 [02mw-config] 07Universal-Omega reviewed pull request 03#5569 commit - 13https://github.com/miraheze/mw-config/pull/5569#discussion_r1611965399
[16:16:24] <botifico>	 [02mw-config] 07brandon-wm synchronize pull request 03#5569: T12111: Add LinkCards to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/5569
[16:16:29] <botifico>	 [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5569: T12111: Add LinkCards to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/5569#issuecomment-2123644505
[16:16:33] <botifico>	 [02mw-config] 07brandon-wm synchronize pull request 03#5569: T12111: Add LinkCards to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/5569
[16:17:27] <miraheze-github>	 miraheze/mw-config - brandon-wm the build passed.
[16:17:28] <miraheze-github_>	 miraheze/mw-config - brandon-wm the build passed.
[16:23:53] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[16:23:54] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[16:23:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:24:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:27:10] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 145.10 ms
[16:29:12] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 145.29 ms
[16:32:33] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[16:32:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:32:43] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s
[16:32:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:42:32] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 142.23 ms
[16:44:33] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 150.01 ms
[17:46:34] <icinga-miraheze>	 PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.barengreza.my.id All nameservers failed to answer the query.
[18:16:33] <icinga-miraheze>	 PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.barengreza.my.id could not be found
[18:42:05] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[19:12:09] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[19:23:03] <icinga-miraheze>	 PROBLEM - segawiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:24:33] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:28:30] <icinga-miraheze>	 PROBLEM - wiki.hwfib.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:29:26] <icinga-miraheze>	 PROBLEM - www.puzzles.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:31:42] <icinga-miraheze>	 PROBLEM - wiki.queenscourt.games - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:36:00] <icinga-miraheze>	 PROBLEM - www.durawiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:39:20] <icinga-miraheze>	 PROBLEM - www.sekaipedia.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:39:51] <icinga-miraheze>	 PROBLEM - wiki.grab-tools.live - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:39:57] <icinga-miraheze>	 PROBLEM - www.programming.red - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:40:37] <icinga-miraheze>	 PROBLEM - bushcraftwiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:42:08] <icinga-miraheze>	 PROBLEM - psychevos.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:42:12] <icinga-miraheze>	 PROBLEM - en.noblework.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:44:21] <icinga-miraheze>	 PROBLEM - wiki.thelastdimension.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:44:48] <icinga-miraheze>	 PROBLEM - wiki.cutefame.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cutefame.net' expires in 15 day(s) (Sat 08 Jun 2024 07:27:49 PM GMT +0000).
[19:45:00] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f62ac07b62ee...16e0951ad09c
[19:45:03] <botifico>	 [02ssl] 07WikiTideSSLBot 0316e0951 - Bot: Update SSL cert for wiki.cutefame.net
[19:46:44] <icinga-miraheze>	 PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:47:26] <icinga-miraheze>	 PROBLEM - de.berlinwiki.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:49:00] <icinga-miraheze>	 PROBLEM - wiki.cube-conflict.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:50:15] <icinga-miraheze>	 PROBLEM - wiki.omegabuild.uk - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:54:03] <icinga-miraheze>	 PROBLEM - wiki.grab-tools.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grab-tools.live' expires in 15 day(s) (Sat 08 Jun 2024 07:26:37 PM GMT +0000).
[19:54:15] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/16e0951ad09c...5a5dbbda17f6
[19:54:18] <botifico>	 [02ssl] 07WikiTideSSLBot 035a5dbbd - Bot: Update SSL cert for wiki.grab-tools.live
[19:56:27] <icinga-miraheze>	 PROBLEM - gogigantic.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gogigantic.wiki' expires in 15 day(s) (Sat 08 Jun 2024 07:39:04 PM GMT +0000).
[19:56:42] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a5dbbda17f6...29c4f5e95a59
[19:56:45] <botifico>	 [02ssl] 07WikiTideSSLBot 0329c4f5e - Bot: Update SSL cert for gogigantic.wiki
[19:56:46] <icinga-miraheze>	 PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[19:59:41] <icinga-miraheze>	 PROBLEM - wiki.pokrew.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:04:54] <icinga-miraheze>	 PROBLEM - metroidpedia.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:05:51] <icinga-miraheze>	 PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:07:53] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]
[20:08:28] <icinga-miraheze>	 RECOVERY - wiki.grab-tools.live - reverse DNS on sslhost is OK: SSL OK - wiki.grab-tools.live reverse DNS resolves to cp37.wikitide.net - CNAME OK
[20:11:33] <icinga-miraheze>	 RECOVERY - psychevos.org - reverse DNS on sslhost is OK: SSL OK - psychevos.org reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[20:11:59] <icinga-miraheze>	 PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[20:12:09] <icinga-miraheze>	 PROBLEM - wiki.elysiamc.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:12:32] <botifico>	 [02RottenLinks] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/03df07623324...b662b7808591
[20:12:34] <botifico>	 [02RottenLinks] 07OAuthority 03b662b78 - consider 410 dead
[20:13:05] <icinga-miraheze>	 PROBLEM - wiki.infomedia.co.uk - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:13:07] <botifico>	 [02RottenLinks] 07OAuthority pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/56e4017c0af0...bdad216d6217
[20:13:08] <botifico>	 [02RottenLinks] 07OAuthority 03bdad216 - consider 410 dead (backport)
[20:14:06] <icinga-miraheze>	 RECOVERY - wiki.cutefame.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cutefame.net' will expire on Wed 21 Aug 2024 06:44:54 PM GMT +0000.
[20:15:32] <icinga-miraheze>	 PROBLEM - programming.red - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:16:52] <miraheze-github>	 miraheze/RottenLinks - OAuthority the build passed.
[20:18:19] <icinga-miraheze>	 PROBLEM - puzzles.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:18:40] <icinga-miraheze>	 PROBLEM - wiki.jill-jimmy.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:23:37] <icinga-miraheze>	 RECOVERY - wiki.grab-tools.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grab-tools.live' will expire on Wed 21 Aug 2024 06:54:09 PM GMT +0000.
[20:24:22] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK
[20:25:24] <icinga-miraheze>	 RECOVERY - gogigantic.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gogigantic.wiki' will expire on Wed 21 Aug 2024 06:56:36 PM GMT +0000.
[20:34:35] <icinga-miraheze>	 RECOVERY - www.durawiki.com - reverse DNS on sslhost is OK: SSL OK - www.durawiki.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[20:35:51] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[20:39:49] <icinga-miraheze>	 PROBLEM - wiki.grab-tools.live - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:39:58] <icinga-miraheze>	 RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[20:41:23] <icinga-miraheze>	 PROBLEM - dkpedia.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:50:52] <icinga-miraheze>	 PROBLEM - www.rippaversewiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[20:53:14] <icinga-miraheze>	 PROBLEM - pandorastale.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pandorastale.wiki' expires in 15 day(s) (Sat 08 Jun 2024 08:37:22 PM GMT +0000).
[20:53:25] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/29c4f5e95a59...c8c2ba71a6d2
[20:53:28] <botifico>	 [02ssl] 07WikiTideSSLBot 03c8c2ba7 - Bot: Update SSL cert for pandorastale.wiki
[20:54:46] <icinga-miraheze>	 PROBLEM - mockgovernments.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mockgovernments.com' expires in 15 day(s) (Sat 08 Jun 2024 08:37:37 PM GMT +0000).
[20:54:57] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c8c2ba71a6d2...9701b021c346
[20:54:58] <botifico>	 [02ssl] 07WikiTideSSLBot 039701b02 - Bot: Update SSL cert for mockgovernments.com
[21:05:42] <icinga-miraheze>	 RECOVERY - adadevelopersacademy.wiki - reverse DNS on sslhost is OK: SSL OK - adadevelopersacademy.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[21:06:06] <icinga-miraheze>	 PROBLEM - www.durawiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[21:08:43] <icinga-miraheze>	 RECOVERY - www.sekaipedia.org - reverse DNS on sslhost is OK: SSL OK - www.sekaipedia.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:10:35] <icinga-miraheze>	 PROBLEM - bushcraftwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - bushcraftwiki.com reverse DNS resolves to any-in-2001-4860-4802-32--15.1e100.net
[21:12:43] <icinga-miraheze>	 RECOVERY - wiki.infomedia.co.uk - reverse DNS on sslhost is OK: SSL OK - wiki.infomedia.co.uk reverse DNS resolves to cp37.wikitide.net - CNAME OK
[21:13:12] <icinga-miraheze>	 RECOVERY - programming.red - reverse DNS on sslhost is OK: SSL OK - programming.red reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK
[21:15:07] <icinga-miraheze>	 RECOVERY - de.berlinwiki.org - reverse DNS on sslhost is OK: SSL OK - de.berlinwiki.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:15:24] <icinga-miraheze>	 RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:16:33] <icinga-miraheze>	 RECOVERY - wiki.jill-jimmy.com - reverse DNS on sslhost is OK: SSL OK - wiki.jill-jimmy.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:16:38] <icinga-miraheze>	 RECOVERY - wiki.cube-conflict.com - reverse DNS on sslhost is OK: SSL OK - wiki.cube-conflict.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:16:49] <icinga-miraheze>	 RECOVERY - puzzles.wiki - reverse DNS on sslhost is OK: SSL OK - puzzles.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK
[21:18:01] <icinga-miraheze>	 RECOVERY - wiki.omegabuild.uk - reverse DNS on sslhost is OK: SSL OK - wiki.omegabuild.uk reverse DNS resolves to cp37.wikitide.net - CNAME OK
[21:20:54] <icinga-miraheze>	 RECOVERY - www.rippaversewiki.com - reverse DNS on sslhost is OK: SSL OK - www.rippaversewiki.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK
[21:22:29] <icinga-miraheze>	 RECOVERY - segawiki.com - reverse DNS on sslhost is OK: SSL OK - segawiki.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[21:23:13] <icinga-miraheze>	 RECOVERY - pandorastale.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pandorastale.wiki' will expire on Wed 21 Aug 2024 07:53:20 PM GMT +0000.
[21:23:31] <icinga-miraheze>	 RECOVERY - mockgovernments.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mockgovernments.com' will expire on Wed 21 Aug 2024 07:54:51 PM GMT +0000.
[21:24:37] <icinga-miraheze>	 RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp37.wikitide.net - CNAME OK
[21:26:10] <icinga-miraheze>	 RECOVERY - wiki.hwfib.com - reverse DNS on sslhost is OK: SSL OK - wiki.hwfib.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:27:04] <icinga-miraheze>	 RECOVERY - www.puzzles.wiki - reverse DNS on sslhost is OK: SSL OK - www.puzzles.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[21:27:06] <icinga-miraheze>	 RECOVERY - wiki.pokrew.net - reverse DNS on sslhost is OK: SSL OK - wiki.pokrew.net reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:29:49] <icinga-miraheze>	 RECOVERY - wiki.queenscourt.games - reverse DNS on sslhost is OK: SSL OK - wiki.queenscourt.games reverse DNS resolves to cp37.wikitide.net - CNAME OK
[21:32:34] <icinga-miraheze>	 RECOVERY - metroidpedia.com - reverse DNS on sslhost is OK: SSL OK - metroidpedia.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK
[21:35:21] <icinga-miraheze>	 RECOVERY - www.durawiki.com - reverse DNS on sslhost is OK: SSL OK - www.durawiki.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[21:37:33] <icinga-miraheze>	 RECOVERY - wiki.grab-tools.live - reverse DNS on sslhost is OK: SSL OK - wiki.grab-tools.live reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:37:37] <icinga-miraheze>	 RECOVERY - www.programming.red - reverse DNS on sslhost is OK: SSL OK - www.programming.red reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[21:39:04] <icinga-miraheze>	 RECOVERY - dkpedia.com - reverse DNS on sslhost is OK: SSL OK - dkpedia.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK
[21:39:46] <icinga-miraheze>	 RECOVERY - wiki.elysiamc.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.elysiamc.xyz reverse DNS resolves to cp37.wikitide.net - CNAME OK
[21:40:02] <icinga-miraheze>	 RECOVERY - en.noblework.org - reverse DNS on sslhost is OK: SSL OK - en.noblework.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:42:06] <icinga-miraheze>	 RECOVERY - wiki.thelastdimension.net - reverse DNS on sslhost is OK: SSL OK - wiki.thelastdimension.net reverse DNS resolves to cp37.wikitide.net - CNAME OK
[21:42:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:07:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[22:49:20] <icinga-miraheze>	 PROBLEM - wiki.solarconquest.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.solarconquest.net' expires in 15 day(s) (Sat 08 Jun 2024 10:18:09 PM GMT +0000).
[22:49:32] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9701b021c346...7519f7a858d7
[22:49:33] <botifico>	 [02ssl] 07WikiTideSSLBot 037519f7a - Bot: Update SSL cert for wiki.solarconquest.net
[22:57:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:32:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:47:58] <icinga-miraheze>	 RECOVERY - wiki.solarconquest.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.solarconquest.net' will expire on Wed 21 Aug 2024 09:49:25 PM GMT +0000.