[00:02:07] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 150.81, 59.00, 21.87 [00:03:47] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 5.71, 5.35, 2.73 [00:07:46] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 1.75, 3.61, 2.66 [00:09:46] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 2.44, 3.40, 2.71 [00:13:45] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 10.88, 7.55, 4.56 [00:22:16] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:25:42] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.61, 2.87, 3.91 [00:29:41] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.47, 1.52, 3.11 [00:33:58] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.45, 1.27, 3.85 [00:37:58] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.41, 0.81, 3.07 [01:50:16] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:23:49] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:53:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:03:32] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.0122423172 secs [03:11:47] PROBLEM - tudien.como.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tudien.como.vn All nameservers failed to answer the query. [03:17:21] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.005881518126 secs [03:41:47] PROBLEM - tudien.como.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tudien.como.vn could not be found [04:27:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.25, 7.30, 4.94 [04:29:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.03, 8.03, 5.42 [04:31:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 7.15, 7.41, 5.50 [04:33:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.43, 10.73, 6.89 [04:37:19] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7162 MB (5% inode=99%); [04:47:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.76, 5.94, 7.68 [04:49:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.62, 7.83, 8.30 [04:57:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.43, 5.81, 7.57 [04:57:37] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [04:59:19] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15031 MB (11% inode=99%); [04:59:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 23.15, 11.46, 9.37 [05:01:37] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [05:02:45] RECOVERY - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chouverse.cludisciples.cyou' will expire on Fri 06 Oct 2023 12:56:23 GMT +0000. [05:05:19] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13014 MB (9% inode=99%); [05:11:47] PROBLEM - tudien.como.vn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tudien.como.vn All nameservers failed to answer the query. [05:19:19] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 16248 MB (12% inode=99%); [05:21:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.89, 5.61, 7.61 [05:25:28] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.24, 2.83, 6.02 [05:34:39] PROBLEM - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.chouverse.cludisciples.cyou and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:41:47] PROBLEM - tudien.como.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tudien.como.vn could not be found [06:13:32] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1011354029 secs [06:15:20] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.99, 3.51, 2.98 [06:24:59] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.32, 4.25, 3.52 [06:27:36] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 5.38, 4.71, 2.34 [06:27:58] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 6.20, 6.50, 3.08 [06:29:36] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 1.28, 3.47, 2.18 [06:30:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.76, 3.90, 3.61 [06:31:36] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 1.94, 2.85, 2.10 [06:31:58] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.71, 3.24, 2.51 [06:38:30] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.25, 3.96, 3.65 [06:40:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.28, 3.90, 3.69 [06:42:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 3.98, 4.09, 3.79 [06:46:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.65, 3.56, 3.67 [06:56:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.16, 3.66, 3.65 [06:58:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.75, 3.69 [07:02:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.01, 2.81, 3.33 [07:19:17] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 15 day(s) (Sat 16 Sep 2023 06:49:45 GMT +0000). [07:22:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c4fee66827a8...823dda6cde6a [07:22:23] [02miraheze/ssl] 07MirahezeSSLBot 03823dda6 - Bot: Update SSL cert for wonderfuleveryday.org [08:17:56] RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Wed 29 Nov 2023 06:22:12 GMT +0000. [08:25:37] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [08:29:37] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [09:11:41] PROBLEM - db142 Backups SQL on db142 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864132 seconds old and 87052 bytes [11:14:18] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/a1e21fa4bf70...0381d67eb40c [11:14:20] [02miraheze/ErrorPages] 07translatewiki 030381d67 - Localisation updates from https://translatewiki.net. [11:14:21] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/4c26dea1718e...ca3db2aee2ca [11:14:22] [02miraheze/GlobalNewFiles] 07translatewiki 03ca3db2a - Localisation updates from https://translatewiki.net. [11:14:24] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/61a1e523ac80...4df4d16c9c20 [11:14:27] [02miraheze/ImportDump] 07translatewiki 034df4d16 - Localisation updates from https://translatewiki.net. [11:14:29] [02miraheze/PDFEmbed] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/b753ff75ed96...e1faaa5b5043 [11:14:31] [02miraheze/PDFEmbed] 07translatewiki 03e1faaa5 - Localisation updates from https://translatewiki.net. [11:14:32] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/67ef731aaf07...16baaf94a408 [11:14:35] [02miraheze/ManageWiki] 07translatewiki 0316baaf9 - Localisation updates from https://translatewiki.net. [11:14:38] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/21ea7585ce8c...e2d0b7758915 [11:14:40] [02miraheze/RemovePII] 07translatewiki 03e2d0b77 - Localisation updates from https://translatewiki.net. [11:14:42] [02miraheze/RottenLinks] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/6493d8ccc18f...43e96cac2b1a [11:14:45] [02miraheze/RottenLinks] 07translatewiki 0343e96ca - Localisation updates from https://translatewiki.net. [11:14:48] [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/SpriteSheet/compare/779545b78350...6b6a1ae31e3e [11:14:50] [02miraheze/SpriteSheet] 07translatewiki 036b6a1ae - Localisation updates from https://translatewiki.net. [11:14:51] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/fb0b66592a83...f0199ae82bfc [11:14:53] [02miraheze/MirahezeMagic] 07translatewiki 03f0199ae - Localisation updates from https://translatewiki.net. [11:14:54] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/01f5e4c62385...44638cd0f42e [11:14:56] [02miraheze/WikiDiscover] 07translatewiki 0344638cd - Localisation updates from https://translatewiki.net. [11:15:21] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.1007337272 secs [11:20:00] miraheze/ErrorPages - translatewiki the build passed. [11:20:28] miraheze/RemovePII - translatewiki the build has errored. [11:20:36] miraheze/PDFEmbed - translatewiki the build passed. [11:21:10] miraheze/RottenLinks - translatewiki the build passed. [11:21:45] miraheze/ManageWiki - translatewiki the build passed. [11:22:21] miraheze/GlobalNewFiles - translatewiki the build passed. [11:24:14] miraheze/SpriteSheet - translatewiki the build passed. [11:26:01] miraheze/MirahezeMagic - translatewiki the build has errored. [11:26:46] miraheze/WikiDiscover - translatewiki the build passed. [11:52:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.59, 3.40, 2.88 [11:54:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.16, 2.97, 2.80 [11:56:02] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:08:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.27, 3.58, 2.89 [12:25:27] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:27:37] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [12:31:37] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [13:20:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.77, 3.39, 3.93 [13:28:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.99, 2.37, 3.23 [14:33:59] PROBLEM - bast141 PowerDNS Recursor on bast141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:33:59] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 3.68, 1.89, 0.80 [14:38:07] RECOVERY - bast141 PowerDNS Recursor on bast141 is OK: DNS OK: 0.259 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [14:41:56] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.19, 1.46, 1.21 [15:02:10] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 2.72, 1.66, 1.04 [15:04:06] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 1.51, 1.72, 1.15 [15:06:03] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.31, 1.21, 1.03 [15:32:14] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 1.98, 1.55, 0.88 [15:34:10] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 2.97, 2.00, 1.13 [15:38:03] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 0.63, 1.76, 1.27 [15:40:01] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.36, 1.23, 1.14 [18:12:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.45, 3.21, 2.61 [18:14:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.82, 3.72, 2.86 [18:16:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.87, 3.75, 2.98 [18:22:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.14, 3.37, 3.05 [18:25:09] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:26:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.18, 3.47, 3.18 [18:32:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.61, 3.34 [18:34:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.73, 3.73, 3.42 [18:38:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.66, 3.96, 3.57 [18:46:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.92, 3.78 [18:54:34] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:56:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.41, 3.78, 3.69 [19:00:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.03, 3.56, 3.64 [19:04:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.21, 2.73, 3.27 [19:23:02] PROBLEM - swiftac111 APT on swiftac111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:25:03] PROBLEM - swiftac111 APT on swiftac111 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (9 critical updates). [20:02:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.36, 3.42, 3.10 [20:04:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.35, 3.12 [20:10:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.19, 3.49, 3.26 [20:12:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.77, 3.34, 3.24 [20:18:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.65, 3.41 [20:20:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.63, 3.40, 3.35 [20:24:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.70, 3.50 [20:30:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.04, 3.31, 3.39 [20:31:32] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5009358823 secs [20:34:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.35, 3.43, 3.43 [20:38:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.68, 3.52 [20:42:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.13, 3.56, 3.53 [20:46:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.28, 2.94, 3.29 [21:32:09] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [22:02:09] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [22:56:33] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:03:28] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.36, 7.03, 3.29 [23:04:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: LOAD CRITICAL - total load average: 5.35, 3.29, 1.92 [23:06:52] PROBLEM - cp33 Current Load on cp33 is WARNING: LOAD WARNING - total load average: 3.35, 3.66, 2.25 [23:08:52] RECOVERY - cp33 Current Load on cp33 is OK: LOAD OK - total load average: 1.54, 2.80, 2.10 [23:09:28] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.55, 7.77, 5.24 [23:11:28] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.79, 5.33, 4.64 [23:25:58] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [23:53:12] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.