[00:00:23] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:01:11] PROBLEM - wiki.ate42.ru - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ate42.ru All nameservers failed to answer the query. [00:10:03] PROBLEM - wiki.junkstore.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.junkstore.xyz All nameservers failed to answer the query. [00:12:58] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:14:29] PROBLEM - wiki.maribelhearn.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.maribelhearn.com' expires in 15 day(s) (Sun 08 Sep 2024 11:47:51 PM GMT +0000). [00:14:40] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/649e0fc0e271...46ea563bb2e9 [00:14:41] [02ssl] 07WikiTideSSLBot 0346ea563 - Bot: Update SSL cert for wiki.maribelhearn.com [00:19:24] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.57, 20.58, 19.35 [00:21:20] RECOVERY - wiki.ate42.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ate42.ru' will expire on Thu 21 Nov 2024 10:53:38 PM GMT +0000. [00:21:42] PROBLEM - tno.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tno.wiki All nameservers failed to answer the query. [00:23:03] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.astralprojections.org All nameservers failed to answer the query. [00:27:21] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.32, 22.40, 20.60 [00:29:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.31, 20.85, 20.29 [00:30:35] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: SSL OK - wiki.nowchess.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:31:19] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mobilityengineer.com All nameservers failed to answer the query. [00:32:19] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wikimedia.cat All nameservers failed to answer the query. [00:32:24] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.joust.ro All nameservers failed to answer the query. [00:32:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:33:40] PROBLEM - gufengcheng.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gufengcheng.top All nameservers failed to answer the query. [00:35:17] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.38, 19.29, 20.04 [00:43:12] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.52, 21.17, 20.52 [00:43:27] RECOVERY - wiki.maribelhearn.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.maribelhearn.com' will expire on Thu 21 Nov 2024 11:16:04 PM GMT +0000. [00:49:09] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.37, 19.89, 20.22 [00:50:42] RECOVERY - tno.wiki - reverse DNS on sslhost is OK: SSL OK - tno.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:52:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:53:03] RECOVERY - wiki.astralprojections.org - reverse DNS on sslhost is OK: SSL OK - wiki.astralprojections.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:53:07] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.66, 20.49, 20.47 [00:55:06] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.14, 20.13, 20.34 [00:59:20] RECOVERY - wiki.ate42.ru - reverse DNS on sslhost is OK: SSL OK - wiki.ate42.ru reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:02:19] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:02:31] RECOVERY - gufengcheng.top - reverse DNS on sslhost is OK: SSL OK - gufengcheng.top reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:08:58] PROBLEM - wiki.eggsdstudios.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eggsdstudios.com All nameservers failed to answer the query. [01:12:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:13:40] PROBLEM - vise.dayid.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vise.dayid.org All nameservers failed to answer the query. [01:14:18] PROBLEM - wiki.digitalcandela.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.digitalcandela.com All nameservers failed to answer the query. [01:14:30] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tmyt105.leyhp.com All nameservers failed to answer the query. [01:15:33] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:17:33] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.villagecollaborative.net All nameservers failed to answer the query. [01:19:20] PROBLEM - wiki.sheepservermc.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sheepservermc.net All nameservers failed to answer the query. [01:19:53] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.50, 19.52, 18.44 [01:21:01] PROBLEM - minescape.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - minescape.wiki All nameservers failed to answer the query. [01:21:46] PROBLEM - tno.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tno.wiki All nameservers failed to answer the query. [01:21:52] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.58, 21.86, 19.42 [01:22:43] PROBLEM - puritwiki.p-e.kr - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.408 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [01:23:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.38, 22.03, 19.79 [01:23:59] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:25:06] PROBLEM - wiki.thesimswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thesimswiki.com All nameservers failed to answer the query. [01:25:06] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.astralprojections.org All nameservers failed to answer the query. [01:25:38] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:27:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:27:49] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.74, 23.03, 20.61 [01:29:48] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.72, 22.79, 20.82 [01:29:49] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.thesimswiki.com All nameservers failed to answer the query. [01:30:22] PROBLEM - wiki.ate42.ru - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ate42.ru All nameservers failed to answer the query. [01:32:38] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nowchess.org All nameservers failed to answer the query. [01:33:24] PROBLEM - gufengcheng.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gufengcheng.top All nameservers failed to answer the query. [01:33:46] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.16, 20.19, 20.15 [01:34:16] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wikimedia.cat All nameservers failed to answer the query. [01:34:24] PROBLEM - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [01:35:29] PROBLEM - wiki.corgicam.tv - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.corgicam.tv All nameservers failed to answer the query. [01:38:00] RECOVERY - wiki.junkstore.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.junkstore.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:42:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:42:44] RECOVERY - www.kinitopedia.lol - reverse DNS on sslhost is OK: SSL OK - www.kinitopedia.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:44:00] RECOVERY - wiki.digitalcandela.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitalcandela.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:54:23] PROBLEM - wiki.thesimswiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [01:58:53] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [02:02:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:07:26] RECOVERY - wiki.eggsdstudios.com - reverse DNS on sslhost is OK: SSL OK - wiki.eggsdstudios.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:10:32] https://cdn.discordapp.com/attachments/808001911868489748/1276725300897517662/Screenshot_20240818_021625_WhatsApp.jpg?ex=66ca9298&is=66c94118&hm=2b4ada9ea13c9a8ddd4ff9a507fefae68c1350b02e939ad71ec05497b9d73d04& [02:10:58] PROBLEM - ru-teirailway.f5.si - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ru-teirailway.f5.si All nameservers failed to answer the query. [02:12:19] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:12:35] @notaracham it's still in this channel btw [02:12:48] got it [02:12:52] 👍 [02:13:22] PROBLEM - wiki.case-clicker.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.case-clicker.com All nameservers failed to answer the query. [02:13:54] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:14:18] PROBLEM - wiki.stag.lol - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.stag.lol All nameservers failed to answer the query. [02:14:31] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Sat 19 Oct 2024 02:09:03 PM GMT +0000. [02:14:51] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: SSL OK - wiki.villagecollaborative.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:21:17] PROBLEM - puritwiki.p-e.kr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - puritwiki.p-e.kr All nameservers failed to answer the query. [02:22:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.13, 21.39, 19.45 [02:23:34] PROBLEM - wiki.thesimswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thesimswiki.com All nameservers failed to answer the query. [02:24:35] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.28, 22.66, 20.14 [02:26:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.43, 21.93, 20.17 [02:27:54] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.thesimswiki.com All nameservers failed to answer the query. [02:28:35] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 23.69, 21.06 [02:32:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.39, 22.65, 21.22 [02:37:53] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [02:38:41] PROBLEM - wiki.eggsdstudios.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.eggsdstudios.com All nameservers failed to answer the query. [02:40:09] PROBLEM - wiki.junkstore.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.junkstore.xyz All nameservers failed to answer the query. [02:40:22] RECOVERY - ru-teirailway.f5.si - reverse DNS on sslhost is OK: SSL OK - ru-teirailway.f5.si reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:41:47] PROBLEM - exordium.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'exordium.wiki' expires in 15 day(s) (Mon 09 Sep 2024 02:22:50 AM GMT +0000). [02:41:53] PROBLEM - wiki.yahia.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahia.xyz All nameservers failed to answer the query. [02:42:49] RECOVERY - wiki.case-clicker.com - reverse DNS on sslhost is OK: SSL OK - wiki.case-clicker.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:43:06] RECOVERY - wiki.stag.lol - reverse DNS on sslhost is OK: SSL OK - wiki.stag.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:43:41] PROBLEM - vise.dayid.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - vise.dayid.org All nameservers failed to answer the query. [02:44:39] PROBLEM - www.kinitopedia.lol - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.kinitopedia.lol All nameservers failed to answer the query. [02:45:28] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.villagecollaborative.net All nameservers failed to answer the query. [02:45:31] PROBLEM - wiki.digitalcandela.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.digitalcandela.com All nameservers failed to answer the query. [02:45:39] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tmyt105.leyhp.com All nameservers failed to answer the query. [02:46:05] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [02:47:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:48:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.33, 19.33, 20.23 [02:51:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:55:21] PROBLEM - rainverse.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rainverse.wiki' expires in 15 day(s) (Mon 09 Sep 2024 02:37:56 AM GMT +0000). [02:55:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/46ea563bb2e9...0ae334b73048 [02:55:36] [02ssl] 07WikiTideSSLBot 030ae334b - Bot: Update SSL cert for rainverse.wiki [02:58:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.36, 19.81, 19.53 [03:00:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.34, 19.31, 19.41 [03:01:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:22:11] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 09 Oct 2024 09:20:12 PM GMT +0000. [03:22:34] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 10 Oct 2024 07:43:36 AM GMT +0000. [03:22:38] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 4.74, 2.97 [03:25:10] RECOVERY - rainverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rainverse.wiki' will expire on Fri 22 Nov 2024 01:56:56 AM GMT +0000. [03:25:54] RECOVERY - www.thesimswiki.com - reverse DNS on sslhost is OK: SSL OK - www.thesimswiki.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:28:29] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.91, 3.87, 3.16 [03:32:22] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.04, 3.09, 2.97 [03:32:39] RECOVERY - wiki.corgicam.tv - reverse DNS on sslhost is OK: SSL OK - wiki.corgicam.tv reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:33:30] RECOVERY - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tmyt105.leyhp.com' will expire on Thu 12 Sep 2024 12:27:49 PM GMT +0000. [03:34:14] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:35:01] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:19] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.89, 5.65, 4.06 [03:36:59] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [03:37:29] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:42:29] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:44:06] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.30, 3.36, 3.62 [03:45:01] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Sat 19 Oct 2024 02:09:03 PM GMT +0000. [03:46:03] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.38, 4.56, 4.03 [03:46:52] RECOVERY - tno.wiki - reverse DNS on sslhost is OK: SSL OK - tno.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:48:00] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.18, 3.73, 3.78 [03:51:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.23, 4.15, 3.89 [03:53:44] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [03:55:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.27, 3.73, 3.79 [03:56:57] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.thesimswiki.com All nameservers failed to answer the query. [03:57:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.69, 4.09, 3.91 [04:01:32] RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: SSL OK - wiki.joust.ro reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:05:07] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Wed 09 Oct 2024 09:25:14 PM GMT +0000. [04:11:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.22, 3.28, 3.95 [04:13:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.66, 4.50, 4.31 [04:14:44] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:16:33] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [04:17:55] PROBLEM - tno.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tno.wiki All nameservers failed to answer the query. [04:18:42] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:21:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.85, 3.35, 3.94 [04:22:58] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 20.76, 19.00 [04:23:16] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 09 Oct 2024 09:20:12 PM GMT +0000. [04:23:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.95, 4.19, 4.19 [04:24:26] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:24:52] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:24:57] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.24, 19.88, 18.88 [04:26:54] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [04:27:29] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:32:29] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:32:50] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.096 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:33:23] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.joust.ro All nameservers failed to answer the query. [04:34:11] PROBLEM - wiki.corgicam.tv - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.corgicam.tv All nameservers failed to answer the query. [04:34:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.72, 20.08, 19.28 [04:35:38] RECOVERY - wiki.eggsdstudios.com - reverse DNS on sslhost is OK: SSL OK - wiki.eggsdstudios.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:37:29] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:39:15] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:40:49] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.91, 19.80, 19.59 [04:41:13] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.065 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:42:29] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:42:39] RECOVERY - wiki.sheepservermc.net - reverse DNS on sslhost is OK: SSL OK - wiki.sheepservermc.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:45:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.88, 3.28, 3.93 [04:46:03] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Sat 19 Oct 2024 02:09:03 PM GMT +0000. [04:46:55] RECOVERY - tno.wiki - reverse DNS on sslhost is OK: SSL OK - tno.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:49:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.81, 4.03, 4.06 [04:51:36] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:53:47] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [04:55:42] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:55:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.29, 3.67, 3.97 [04:57:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 4.03, 4.04 [05:00:08] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:00:33] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: SSL OK - wiki.nowchess.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:01:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.42, 18.51, 23.21 [05:02:07] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.099 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:05:15] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 22.07, 23.46 [05:06:22] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:33] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:07:11] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wikimedia.cat All nameservers failed to answer the query. [05:07:41] PROBLEM - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [05:10:18] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:10:19] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [05:10:40] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.095 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:11:20] PROBLEM - spiceandwolf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spiceandwolf.wiki' expires in 15 day(s) (Mon 09 Sep 2024 04:43:43 AM GMT +0000). [05:11:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ae334b73048...40933093d20b [05:11:40] [02ssl] 07WikiTideSSLBot 034093309 - Bot: Update SSL cert for spiceandwolf.wiki [05:12:29] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:13:40] PROBLEM - wiki.sheepservermc.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sheepservermc.net All nameservers failed to answer the query. [05:14:07] RECOVERY - www.kinitopedia.lol - reverse DNS on sslhost is OK: SSL OK - www.kinitopedia.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:15:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:17:57] PROBLEM - tno.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tno.wiki All nameservers failed to answer the query. [05:21:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.30, 3.26, 3.85 [05:22:59] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 10 Oct 2024 07:43:36 AM GMT +0000. [05:24:27] RECOVERY - gufengcheng.top - reverse DNS on sslhost is OK: SSL OK - gufengcheng.top reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:25:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.42, 3.91, 3.95 [05:27:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.10, 3.30, 3.74 [05:29:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.45, 4.05, 3.95 [05:31:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.26, 3.47, 3.77 [05:32:35] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nowchess.org All nameservers failed to answer the query. [05:35:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.48, 3.67 [05:36:27] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o