[00:01:38] PROBLEM - changeprop151 Disk Space on changeprop151 is WARNING: DISK WARNING - free space: / 1748MiB (9% inode=89%); [00:21:54] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [00:33:45] [02mw-config] 07BlankEclair opened pull request 03#5604: T12308: Add *.sm2d, *.smis, and *.smms - 13https://github.com/miraheze/mw-config/pull/5604 [00:34:42] miraheze/mw-config - BlankEclair the build passed. [02:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:49:31] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:03:38] PROBLEM - changeprop151 Disk Space on changeprop151 is CRITICAL: DISK CRITICAL - free space: / 1080MiB (5% inode=89%); [04:41:09] PROBLEM - tfuk.app - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tfuk.app' expires in 15 day(s) (Fri 26 Jul 2024 04:37:28 AM GMT +0000). [04:41:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c683d804c434...991880c7a126 [04:41:23] [02ssl] 07WikiTideSSLBot 03991880c - Bot: Update SSL cert for tfuk.app [04:52:59] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [05:09:49] RECOVERY - tfuk.app - LetsEncrypt on sslhost is OK: OK - Certificate 'tfuk.app' will expire on Tue 08 Oct 2024 03:41:14 AM GMT +0000. [05:53:31] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.32, 20.86, 18.47 [05:57:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.27, 19.59, 18.54 [06:14:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.74, 20.87, 19.56 [06:16:29] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.42, 20.26, 19.49 [06:21:16] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:13] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008313953876 secs [06:25:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 21.74, 20.19 [06:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.73, 22.09, 20.78 [06:30:40] PROBLEM - dlfm-wiki.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dlfm-wiki.top All nameservers failed to answer the query. [06:33:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.53, 22.38, 21.11 [06:34:34] PROBLEM - www.streamerlore.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.streamerlore.com All nameservers failed to answer the query. [06:35:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.23, 21.20, 20.86 [06:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 21.75, 21.12 [06:45:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.27, 20.83, 20.97 [06:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:51:37] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:55:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.18, 22.59, 21.79 [06:57:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.88, 23.30, 22.18 [07:00:18] RECOVERY - dlfm-wiki.top - reverse DNS on sslhost is OK: SSL OK - dlfm-wiki.top reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:04:09] RECOVERY - www.streamerlore.com - reverse DNS on sslhost is OK: SSL OK - www.streamerlore.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:05:13] PROBLEM - with.cpt-ra.bid - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [07:11:08] PROBLEM - history.estill.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - history.estill.org All nameservers failed to answer the query. [07:11:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 22.14, 21.66 [07:12:06] PROBLEM - wiki.fediverse.kr - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.fediverse.kr All nameservers failed to answer the query. [07:12:34] PROBLEM - wiki.zergmafia.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zergmafia.top All nameservers failed to answer the query. [07:12:57] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [07:13:00] PROBLEM - wiki.pulsus.cc - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.pulsus.cc All nameservers failed to answer the query. [07:15:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.77, 22.04, 21.77 [07:15:39] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:19:44] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0006691217422 secs [07:33:46] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Tue 06 Aug 2024 02:45:02 PM GMT +0000. [07:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.69, 24.04, 22.52 [07:39:44] RECOVERY - history.estill.org - reverse DNS on sslhost is OK: SSL OK - history.estill.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:41:05] PROBLEM - amps.wiki.gd - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - amps.wiki.gd All nameservers failed to answer the query. [07:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.36, 22.69, 22.33 [07:41:55] RECOVERY - wiki.altilunium.my.id - reverse DNS on sslhost is OK: SSL OK - wiki.altilunium.my.id reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:42:03] RECOVERY - wiki.fediverse.kr - reverse DNS on sslhost is OK: SSL OK - wiki.fediverse.kr reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:42:26] RECOVERY - wiki.zergmafia.top - reverse DNS on sslhost is OK: SSL OK - wiki.zergmafia.top reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:42:29] RECOVERY - wiki.pulsus.cc - reverse DNS on sslhost is OK: SSL OK - wiki.pulsus.cc reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:02:29] PROBLEM - wiki.gamergeeked.us - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gamergeeked.us All nameservers failed to answer the query. [08:03:59] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query. [08:07:20] PROBLEM - minescape.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - minescape.wiki All nameservers failed to answer the query. [08:07:29] PROBLEM - www.kinitopedia.lol - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.kinitopedia.lol All nameservers failed to answer the query. [08:08:04] PROBLEM - wiki.yahia.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahia.xyz All nameservers failed to answer the query. [08:09:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 22.93, 22.20 [08:09:40] RECOVERY - amps.wiki.gd - reverse DNS on sslhost is OK: SSL OK - amps.wiki.gd reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:11:56] PROBLEM - lab.wikimimo.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lab.wikimimo.com All nameservers failed to answer the query. [08:12:14] PROBLEM - tfuk.app - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tfuk.app All nameservers failed to answer the query. [08:13:23] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [08:17:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.75, 23.15, 22.94 [08:27:58] [02ssl] 07BlankEclair opened pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788 [08:28:04] [02ssl] 07coderabbitai[bot] commented on pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788#issuecomment-2219879002 [08:29:11] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788#issuecomment-2219879002 [08:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.47, 24.40, 23.09 [08:31:29] RECOVERY - wiki.gamergeeked.us - reverse DNS on sslhost is OK: SSL OK - wiki.gamergeeked.us reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:33:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.66, 22.74, 22.84 [08:34:00] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'} [08:36:53] RECOVERY - minescape.wiki - reverse DNS on sslhost is OK: SSL OK - minescape.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT [08:36:58] RECOVERY - www.kinitopedia.lol - reverse DNS on sslhost is OK: SSL OK - www.kinitopedia.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 23.91, 23.20 [08:38:00] RECOVERY - wiki.yahia.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.yahia.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:39:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.64, 21.90, 22.55 [08:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 23.13, 22.93 [08:41:22] RECOVERY - lab.wikimimo.com - reverse DNS on sslhost is OK: SSL OK - lab.wikimimo.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:42:07] RECOVERY - tfuk.app - reverse DNS on sslhost is OK: SSL OK - tfuk.app reverse DNS resolves to cp37.wikitide.net - CNAME FLAT [08:42:33] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT [08:42:39] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:43:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.28, 22.46, 22.69 [08:52:25] PROBLEM - wiki.widwwa.co.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.widwwa.co.uk All nameservers failed to answer the query. [08:55:03] PROBLEM - www.fossilfighterswiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.fossilfighterswiki.com All nameservers failed to answer the query. [09:11:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.14, 17.98, 20.02 [09:12:15] RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Sun 28 Jul 2024 03:38:42 AM GMT +0000. [09:21:47] RECOVERY - wiki.widwwa.co.uk - reverse DNS on sslhost is OK: SSL OK - wiki.widwwa.co.uk reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:25:03] RECOVERY - www.fossilfighterswiki.com - reverse DNS on sslhost is OK: SSL OK - www.fossilfighterswiki.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:34:29] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.05, 20.95, 20.04 [09:38:17] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.78, 18.99, 19.52 [10:41:26] PROBLEM - db151 Backups SQL on db151 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209675 seconds old and 134446 bytes [10:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:16:51] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:14:51] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:45:17] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Sun 28 Jul 2024 02:37:10 AM GMT +0000. [14:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:49:36] RECOVERY - archive.a2b2.org - reverse DNS on sslhost is OK: SSL OK - archive.a2b2.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:56:42] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:50] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o