[00:02:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 13.42, 7.78, 3.41 [00:03:14] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 5.87, 2.86 [00:06:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.96, 3.96, 2.83 [00:08:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.75, 2.99, 2.62 [00:16:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 3.88, 3.74, 2.97 [00:18:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 6.52, 4.96, 3.50 [00:20:14] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 1.52, 3.57, 3.17 [00:21:28] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.10, 3.58, 2.11 [00:22:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 6.08, 4.59, 3.58 [00:23:28] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.84, 3.73, 2.36 [00:24:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.05, 3.14, 3.16 [00:27:28] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.69, 2.98, 2.40 [00:31:14] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.22, 1.59, 3.94 [00:35:14] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.40, 1.05, 3.19 [00:42:42] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14610 MB (10% inode=99%); [02:27:53] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:12] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:30:57] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:31:32] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:31:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:31:38] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:33:22] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:35:19] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.006946206093 secs [02:35:45] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 3957MiB (55% inode=83%); [02:42:38] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 2.87, 2.16 [03:13:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.31, 2.95, 2.28 [03:17:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.54, 3.27, 2.57 [03:19:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.18, 3.22, 2.64 [03:27:32] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:56:05] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 2.73, 2.15 [04:15:18] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 7.59, 8.90, 5.81 [04:16:42] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 6982 MB (5% inode=99%); [04:17:13] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.23, 7.00, 5.48 [04:17:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.45, 2.65, 2.37 [04:19:07] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.29, 5.43, 5.08 [04:26:43] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 9.53, 8.27, 6.29 [04:28:37] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.25, 6.28, 5.79 [04:34:42] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 16235 MB (12% inode=99%); [04:40:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.71, 13.52, 9.01 [04:42:42] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13612 MB (10% inode=99%); [04:54:42] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 17383 MB (13% inode=99%); [05:00:30] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.53, 3.83, 7.30 [05:02:30] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.53, 3.26, 6.65 [06:11:19] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1207947433 secs [06:28:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 6.51, 3.98, 1.89 [06:30:14] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 1.42, 2.81, 1.71 [06:35:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.46, 3.18 [06:37:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.15, 3.61, 3.26 [06:41:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.79, 3.91, 3.48 [06:47:02] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.93, 3.61 [06:48:59] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.85, 3.95, 3.65 [07:02:32] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.40, 2.81, 3.31 [07:23:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:52:35] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [08:23:32] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [08:52:33] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:01:51] PROBLEM - db142 Backups SQL on db142 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864153 seconds old and 86917 bytes [09:25:46] PROBLEM - wiki.sofurry.games - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sofurry.games' expires in 15 day(s) (Sat 02 Sep 2023 09:21:01 GMT +0000). [09:28:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/534f7bd83ba8...25253baf0491 [09:28:42] [02miraheze/ssl] 07MirahezeSSLBot 0325253ba - Bot: Update SSL cert for wiki.sofurry.games [10:53:02] PROBLEM - cloud13 Puppet on cloud13 is UNKNOWN: NRPE: Unable to read output [11:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.18, 2.76 [11:13:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.51, 3.03, 2.77 [11:16:52] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/848800330eec...e49ba62520e9 [11:16:54] [02miraheze/ImportDump] 07translatewiki 03e49ba62 - Localisation updates from https://translatewiki.net. [11:16:56] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/d37fa5498ce9...45b6ea886568 [11:16:58] [02miraheze/MirahezeMagic] 07translatewiki 0345b6ea8 - Localisation updates from https://translatewiki.net. [11:21:02] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:23:29] miraheze/ImportDump - translatewiki the build has errored. [11:24:04] miraheze/MirahezeMagic - translatewiki the build has errored. [12:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.61, 3.59, 3.11 [12:13:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.97, 3.89, 3.35 [12:15:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.39, 4.01, 3.46 [12:23:32] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:33:51] PROBLEM - wiki.closai.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.closai.com' expires in 15 day(s) (Sat 02 Sep 2023 12:26:27 GMT +0000). [12:35:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/25253baf0491...6a0e7915479b [12:35:34] [02miraheze/ssl] 07MirahezeSSLBot 036a0e791 - Bot: Update SSL cert for wiki.closai.com [12:43:08] [02SpriteSheet] 07KellyFromSales opened issue 03#16: internal_api_error_Error when trying to save spritesheet - 13https://github.com/miraheze/SpriteSheet/issues/16 [12:49:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.70, 3.91 [12:52:33] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.75, 3.87 [12:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.94, 3.86, 3.89 [13:05:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.18, 2.70, 3.37 [13:53:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:11:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.77, 3.23, 2.79 [14:13:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.26, 3.25, 2.86 [14:17:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 3.37, 3.00 [14:19:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.25, 3.01 [14:52:30] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:03:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:21:31] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:21:44] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [15:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [15:35:45] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.008539617062 secs [15:46:29] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.42, 2.98 [15:48:25] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.53, 3.14, 2.94 [16:27:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:56:01] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [17:41:53] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.46, 3.23, 2.71 [17:47:42] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.65, 3.06 [17:49:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.96, 3.70, 3.15 [17:57:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.74, 3.21, 3.17 [18:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.49, 4.14, 3.50 [18:51:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.72, 3.95 [18:53:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.61, 4.02, 4.03 [19:01:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.75, 3.96 [19:07:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.33, 2.44, 3.25 [19:22:37] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:24:43] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 7.532 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:57:40] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:59:45] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2745528519 secs [20:01:40] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [20:25:45] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.5564519167 secs [20:57:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:26:02] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:55:31] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [22:07:14] PROBLEM - wiki.susqu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.susqu.org could not be found [22:10:15] PROBLEM - cp32 Current Load on cp32 is WARNING: LOAD WARNING - total load average: 1.25, 3.41, 2.50 [22:12:15] RECOVERY - cp32 Current Load on cp32 is OK: LOAD OK - total load average: 1.17, 2.66, 2.33 [22:27:31] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:56:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [23:04:30] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.01, 8.53, 3.88 [23:08:30] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.70, 6.39, 4.22 [23:33:11] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:35:18] RECOVERY - wiki.susqu.org - reverse DNS on sslhost is OK: SSL OK - wiki.susqu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [23:50:33] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query publictestwiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:53:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query.