[00:10:11] RECOVERY - wiki.anempireofdreams.com - reverse DNS on sslhost is OK: SSL OK - wiki.anempireofdreams.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:57:15] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 2672MiB (37% inode=83%); [02:04:11] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:24:39] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 2672MiB (37% inode=83%); [02:31:34] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:20:02] PROBLEM - corru.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'corru.wiki' expires in 15 day(s) (Sun 05 Nov 2023 06:48:26 GMT +0000). [07:25:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/23e40e4456dc...d628ca857901 [07:25:52] [02miraheze/ssl] 07MirahezeSSLBot 03d628ca8 - Bot: Update SSL cert for corru.wiki [08:18:02] RECOVERY - corru.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'corru.wiki' will expire on Thu 18 Jan 2024 06:25:42 GMT +0000. [11:33:04] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:35:03] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.293 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [13:51:02] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [13:55:02] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:27:15] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:29:19] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 5.273 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [15:07:01] PROBLEM - wikigenius.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikigenius.org' expires in 15 day(s) (Sun 05 Nov 2023 14:56:12 GMT +0000). [15:08:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d628ca857901...f6773d8d2227 [15:09:02] [02miraheze/ssl] 07MirahezeSSLBot 03f6773d8 - Bot: Update SSL cert for wikigenius.org [15:22:45] PROBLEM - wikiyri.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiyri.org' expires in 15 day(s) (Sun 05 Nov 2023 15:12:28 GMT +0000). [15:23:46] PROBLEM - bn.wikiyri.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bn.wikiyri.org' expires in 15 day(s) (Sun 05 Nov 2023 15:18:52 GMT +0000). [15:23:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f6773d8d2227...04b032297ea0 [15:24:01] [02miraheze/ssl] 07MirahezeSSLBot 0304b0322 - Bot: Update SSL cert for wikiyri.org [15:26:05] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/04b032297ea0...87c6685f410d [15:26:07] [02miraheze/ssl] 07MirahezeSSLBot 0387c6685 - Bot: Update SSL cert for bn.wikiyri.org [15:36:03] RECOVERY - wikigenius.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikigenius.org' will expire on Thu 18 Jan 2024 14:08:52 GMT +0000. [15:52:12] PROBLEM - www.esolibrary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.esolibrary.com' expires in 15 day(s) (Sun 05 Nov 2023 15:35:59 GMT +0000). [15:52:45] PROBLEM - wiki.esufranchise.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.esufranchise.com' expires in 15 day(s) (Sun 05 Nov 2023 15:38:07 GMT +0000). [15:53:03] RECOVERY - bn.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bn.wikiyri.org' will expire on Thu 18 Jan 2024 14:25:58 GMT +0000. [16:00:57] PROBLEM - wiki.theatlas.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.theatlas.live' expires in 15 day(s) (Sun 05 Nov 2023 15:39:10 GMT +0000). [16:03:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/87c6685f410d...d8bfe0007d98 [16:03:23] [02miraheze/ssl] 07MirahezeSSLBot 03d8bfe00 - Bot: Update SSL cert for wiki.esufranchise.com [16:07:09] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.48, 2.81, 1.81 [16:09:09] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.07, 2.45, 1.80 [16:09:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d8bfe0007d98...3e613eb6cb73 [16:09:21] [02miraheze/ssl] 07MirahezeSSLBot 033e613eb - Bot: Update SSL cert for wiki.theatlas.live [16:12:14] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 17.49, 11.11, 6.62 [16:13:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.62, 2.44, 1.22 [16:14:14] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 10.54, 10.37, 6.88 [16:15:09] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.88, 2.19, 1.28 [16:16:13] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 6.10, 8.94, 6.79 [16:19:41] PROBLEM - wariopedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wariopedia.com' expires in 7 day(s) (Sat 28 Oct 2023 16:14:50 GMT +0000). [16:22:13] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Thu 18 Jan 2024 14:23:51 GMT +0000. [16:29:55] PROBLEM - wiki.theatlas.live - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.theatlas.live and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:46:29] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Sun 05 Nov 2023 16:43:26 GMT +0000). [16:52:13] RECOVERY - wiki.esufranchise.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.esufranchise.com' will expire on Thu 18 Jan 2024 15:03:14 GMT +0000. [16:58:57] RECOVERY - wiki.theatlas.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.theatlas.live' will expire on Thu 18 Jan 2024 15:09:12 GMT +0000. [17:03:32] PROBLEM - www.project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 15 day(s) (Sun 05 Nov 2023 16:57:56 GMT +0000). [17:04:05] PROBLEM - project-patterns.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'project-patterns.com' expires in 15 day(s) (Sun 05 Nov 2023 16:57:56 GMT +0000). [17:06:20] PROBLEM - miraheze.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.com' expires in 15 day(s) (Sun 05 Nov 2023 16:40:52 GMT +0000). [17:09:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3e613eb6cb73...d157902788c3 [17:09:19] [02miraheze/ssl] 07MirahezeSSLBot 03d157902 - Bot: Update SSL cert for project-patterns.com [17:17:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d157902788c3...676195c4982d [17:17:32] [02miraheze/ssl] 07MirahezeSSLBot 03676195c - Bot: Update SSL cert for miraheze.com [17:33:10] RECOVERY - project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Thu 18 Jan 2024 16:09:09 GMT +0000. [18:01:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.67, 6.50, 5.81 [18:01:34] RECOVERY - www.project-patterns.com - LetsEncrypt on sslhost is OK: OK - Certificate 'project-patterns.com' will expire on Thu 18 Jan 2024 16:09:09 GMT +0000. [18:03:24] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.45, 5.68, 5.59 [18:03:45] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [18:05:50] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Thu 18 Jan 2024 16:17:22 GMT +0000. [18:25:04] PROBLEM - cloud14 Puppet on cloud14 is UNKNOWN: NRPE: Unable to read output [18:51:01] PROBLEM - thenintendowiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thenintendowiki.com could not be found [18:51:04] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:38:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.31, 5.87, 3.53 [19:42:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.79, 7.18, 4.62 [19:46:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.31, 7.75, 5.44 [19:48:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.50, 7.32, 5.57 [20:01:33] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp25.miraheze.org - CNAME OK [20:06:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.29, 7.66, 6.85 [20:20:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.55, 7.91, 7.70 [20:34:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.15, 7.70, 7.64 [20:48:45] !log [void@mwtask141] resetUserEmail.php for HeavyTF2 [20:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.62, 7.64, 7.90 [21:04:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.95, 7.98, 7.88 [21:11:55] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:13:54] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.418 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [21:24:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 6.89, 7.59, 7.89 [21:27:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.45, 3.01, 2.35 [21:31:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.87, 3.02, 2.51 [21:36:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.61, 7.77, 7.70 [21:43:07] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.1225314736 secs [21:46:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.79, 7.96, 7.90 [21:48:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 7.91, 8.03, 7.94 [21:50:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.77, 7.88, 7.89 [21:59:07] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.05237212777 secs [22:04:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.22, 7.88, 7.83 [22:24:21] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.10, 7.64, 7.96 [22:34:21] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.21, 7.70, 7.78 [23:00:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:04:26] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.86, 8.11, 3.89 [23:10:26] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.33, 8.00, 5.47 [23:12:26] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.91, 5.51, 4.86 [23:24:43] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:25:13] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset 0.141762048 secs [23:28:49] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.254 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [23:29:13] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset -0.03678396344 secs [23:30:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [23:33:19] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset -0.154153347 secs [23:37:19] RECOVERY - jobchron121 NTP time on jobchron121 is OK: NTP OK: Offset 0.08950558305 secs [23:59:25] PROBLEM - jobchron121 NTP time on jobchron121 is WARNING: NTP WARNING: Offset 0.1850459278 secs