[00:16:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.49, 3.66, 3.94 [00:20:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.83, 3.93 [00:22:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.89, 3.95, 3.97 [00:26:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 4.21, 4.08 [00:26:49] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [00:30:49] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:41:56] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.500813514 secs [03:45:48] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tmyt105.leyhp.com All nameservers failed to answer the query. [04:14:24] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [06:22:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.56, 3.62, 3.98 [06:24:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 3.88, 3.82, 4.02 [06:28:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 1.26, 3.06, 3.74 [06:30:10] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 0.90, 2.30, 3.37 [06:36:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 3.75, 3.61 [06:38:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.53, 3.86, 3.68 [06:40:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.41, 4.04, 3.76 [06:54:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.06, 3.80, 3.93 [06:56:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 6.26, 4.61, 4.20 [07:26:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.22, 3.59, 3.93 [07:32:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 4.00, 3.98 [07:34:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.74, 3.97, 3.98 [07:36:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 4.16, 4.04 [07:45:50] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tmyt105.leyhp.com All nameservers failed to answer the query. [07:50:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.75, 3.76, 3.92 [07:56:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.10, 3.84, 3.87 [07:58:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.81, 3.84, 3.87 [08:06:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.79, 4.02, 3.88 [08:12:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.33, 3.87, 3.88 [08:14:26] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [08:16:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 3.94, 3.90 [09:37:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [10:07:14] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [11:41:56] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.004109442234 secs [12:19:56] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1100210547 secs [12:25:56] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5773336291 secs [14:20:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 1.79, 2.43, 3.84 [14:22:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.34, 3.20, 3.95 [14:24:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.16, 3.15, 3.84 [14:26:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.72, 3.97 [14:28:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.38, 3.68, 3.94 [14:30:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 4.05, 4.04 [14:31:56] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.008648991585 secs [15:07:56] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1097049117 secs [15:37:35] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 11.02, 12.95, 7.98 [15:37:46] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.63, 2.96, 2.37 [15:39:34] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.86, 9.55, 7.33 [15:39:46] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.61, 2.52, 2.29 [15:45:52] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4089343547821045 seconds [16:09:29] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.77, 3.20, 1.95 [16:11:24] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.55, 2.56, 1.86 [16:14:28] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [16:22:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.64, 6.91, 6.19 [16:24:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.05, 6.29, 6.05 [16:36:02] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404141426086426 seconds [17:05:23] RECOVERY - elephants.ggreig.com - reverse DNS on sslhost is OK: SSL OK - elephants.ggreig.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [17:33:01] PROBLEM - mw142 Puppet on mw142 is UNKNOWN: NRPE: Unable to read output [17:42:50] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [17:43:38] PROBLEM - fanonpedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanonpedia.com' expires in 15 day(s) (Sun 29 Oct 2023 17:33:54 GMT +0000). [17:47:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/43e4ff7c7c33...6013ce2aef98 [17:47:09] [02miraheze/ssl] 07MirahezeSSLBot 036013ce2 - Bot: Update SSL cert for fanonpedia.com [17:50:58] PROBLEM - gil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gil.wiki' expires in 15 day(s) (Sun 29 Oct 2023 17:45:41 GMT +0000). [17:58:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6013ce2aef98...aae3595d7aae [17:58:18] [02miraheze/ssl] 07MirahezeSSLBot 03aae3595 - Bot: Update SSL cert for gil.wiki [18:01:02] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:10:50] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:16] RECOVERY - fanonpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanonpedia.com' will expire on Thu 11 Jan 2024 16:46:59 GMT +0000. [18:14:36] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [18:33:01] PROBLEM - mw142 Puppet on mw142 is UNKNOWN: NRPE: Unable to read output [18:42:36] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:49:55] RECOVERY - gil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gil.wiki' will expire on Thu 11 Jan 2024 16:58:08 GMT +0000. [18:59:38] PROBLEM - test131 Puppet on test131 is UNKNOWN: NRPE: Unable to read output [19:01:01] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [19:15:53] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406136512756348 seconds [19:27:37] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:38:50] !log [@mwtask141] starting deploy of {'config': True} to all [19:38:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:58] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [19:39:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:37] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [20:10:35] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:43:36] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [21:11:00] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.48, 2.75, 2.22 [21:13:00] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.60, 2.33, 2.13 [21:42:36] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [22:06:50] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [22:10:49] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [22:12:36] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:30:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 1.15, 2.89, 3.96 [22:36:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 3.64, 3.88 [22:38:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.69, 3.72, 3.89 [22:39:23] PROBLEM - mw132 Puppet on mw132 is UNKNOWN: NRPE: Unable to read output [22:40:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.39, 3.97, 3.96 [22:42:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.93, 3.99, 3.97 [22:42:36] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [22:47:01] PROBLEM - matomo121 Disk Space on matomo121 is WARNING: DISK WARNING - free space: / 2080MiB (10% inode=91%); [22:50:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.64, 4.37, 4.10 [23:04:33] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.13, 9.92, 5.97 [23:05:22] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:06:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.93, 3.73, 4.00 [23:06:58] PROBLEM - mw141 Puppet on mw141 is UNKNOWN: NRPE: Unable to read output [23:08:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.85, 4.21, 4.15 [23:10:33] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.23, 6.23, 5.87 [23:10:35] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:32:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.71, 3.73, 4.00 [23:33:01] PROBLEM - mw142 Puppet on mw142 is UNKNOWN: NRPE: Unable to read output [23:34:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.33, 4.33, 4.19 [23:36:58] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures