[00:03:16] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.37, 7.97, 3.64 [00:11:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 8.03, 7.61 [00:13:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 7.89, 7.63 [00:15:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 8.18, 7.76 [00:17:16] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.31, 7.40, 7.21 [00:17:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.87, 7.70 [00:19:16] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.86, 5.82, 6.64 [00:31:36] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.22, 2.45 [00:33:37] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.56, 3.32, 2.59 [00:37:14] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.42, 3.85, 3.04 [00:37:31] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.33, 3.39, 2.79 [00:39:08] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.21, 3.59, 3.04 [00:39:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.38, 6.78 [00:41:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.34, 3.83, 3.19 [00:43:01] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.21, 3.53, 3.16 [00:45:01] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.92, 3.32, 3.12 [00:49:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.56, 7.09, 6.68 [00:51:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.10, 6.56, 6.54 [00:57:27] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=beidipediawiki (END - exit=256) [00:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 7.01, 6.70 [01:26:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.62, 7.03 [01:28:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.31, 6.98 [01:30:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 8.00, 7.27 [01:32:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.58, 7.19 [01:42:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.36, 6.48, 6.76 [01:45:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.00, 6.73, 6.84 [01:47:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.00, 6.91 [01:49:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.23, 7.00 [02:01:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.44, 7.42, 7.10 [02:02:03] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.13, 6.13, 5.63 [02:03:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.99, 6.93, 6.97 [02:04:03] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 6.19, 5.70 [02:05:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.21, 7.70, 7.26 [02:06:03] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.44, 6.01, 5.70 [02:07:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 7.56, 7.26 [02:17:59] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [02:21:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 7.12, 6.99 [02:23:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 6.68, 6.83 [02:25:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.03, 6.55, 6.77 [02:32:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 6.77, 6.82 [02:36:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.57, 6.24, 6.59 [02:40:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.10, 7.29, 6.91 [02:42:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.18, 6.92 [02:44:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.28, 6.44, 6.68 [03:00:49] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 48 seconds old and 0 bytes [03:04:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:07:59] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:09:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 7.02, 6.69 [03:13:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.64, 7.99, 7.10 [03:15:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.97, 7.27, 6.95 [03:16:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.46, 7.66, 7.11 [03:18:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.76, 7.23 [03:32:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.00, 6.23, 6.67 [03:34:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:37:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:41:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 6.79, 6.70 [03:42:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.79, 6.65, 6.65 [03:50:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.52, 6.94 [03:52:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.21, 6.89 [03:56:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.58, 7.10 [03:58:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.37, 7.08 [04:02:11] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.36, 6.45, 6.74 [04:06:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 6.82, 6.83 [04:06:15] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [04:07:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.54, 6.73 [04:11:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 7.03, 6.89 [04:13:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.81, 6.59, 6.75 [04:20:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.13, 8.34, 7.41 [04:22:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.65, 7.28, 7.13 [04:30:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.33, 6.73 [04:49:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.94, 6.60 [04:55:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.16, 6.54, 6.55 [05:44:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [05:48:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [06:11:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 6.84, 6.25 [06:13:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.43, 6.67, 6.26 [06:17:59] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:43:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 7.20, 6.61 [06:45:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.91, 6.42, 6.38 [07:07:59] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 6.90, 6.34 [07:34:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.62, 6.68, 6.32 [07:43:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 6.84, 6.43 [07:45:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.88, 6.40, 6.31 [08:07:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 6.37, 6.17 [08:09:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.15, 6.12 [08:10:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:13:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 7.05, 6.53 [08:17:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.04, 6.27, 6.36 [09:08:12] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:21:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.67, 6.68, 6.34 [09:22:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.22, 6.20 [09:37:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:06:17] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [10:17:59] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [11:07:59] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:34:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:04:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [12:34:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:04:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:35:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.14, 6.51 [13:37:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.84, 6.50, 6.35 [13:37:03] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:06:06] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [14:34:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 6.73, 6.31 [14:36:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.33, 6.50, 6.28 [14:52:46] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.007395297289 secs [15:12:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 6.50, 6.11 [15:28:46] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1080767214 secs [15:30:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.15, 6.84 [15:32:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 7.32, 6.94 [15:37:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.32, 6.46, 6.71 [15:40:29] PROBLEM - elsterclopedie.marithaime.nl - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - elsterclopedie.marithaime.nl All nameservers failed to answer the query. [15:41:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 6.97, 6.92 [15:43:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.50, 6.41, 6.72 [16:07:01] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.96, 2.79, 1.87 [16:09:50] RECOVERY - elsterclopedie.marithaime.nl - reverse DNS on sslhost is OK: SSL OK - elsterclopedie.marithaime.nl reverse DNS resolves to cp25.miraheze.org - CNAME OK [16:11:01] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.77, 2.92, 2.14 [16:21:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 7.08, 6.61 [16:23:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.67, 6.52 [17:02:09] PROBLEM - gil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gil.wiki' expires in 15 day(s) (Thu 11 Jan 2024 16:58:08 GMT +0000). [17:02:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fba8565101db...6c15d140713a [17:03:01] [02miraheze/ssl] 07MirahezeSSLBot 036c15d14 - Bot: Update SSL cert for gil.wiki [17:03:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.42, 2.95, 1.59 [17:04:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:05:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.28, 2.39, 1.56 [17:05:37] PROBLEM - fanonpedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanonpedia.com' expires in 15 day(s) (Thu 11 Jan 2024 16:46:59 GMT +0000). [17:05:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6c15d140713a...595371320a4a [17:05:52] [02miraheze/ssl] 07MirahezeSSLBot 035953713 - Bot: Update SSL cert for fanonpedia.com [17:17:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 6.95, 6.68 [17:21:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.44, 6.43, 6.54 [17:30:48] RECOVERY - gil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gil.wiki' will expire on Mon 25 Mar 2024 16:02:27 GMT +0000. [17:31:54] PROBLEM - wiki.hsdfz.org.cn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.hsdfz.org.cn All nameservers failed to answer the query. [17:32:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.71, 7.48, 6.87 [17:34:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:34:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.77, 6.68 [17:48:46] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5003550947 secs [17:51:16] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 7.89, 5.53, 4.36 [17:53:16] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.87, 5.43, 4.48 [18:01:55] PROBLEM - wiki.hsdfz.org.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hsdfz.org.cn could not be found [18:03:50] RECOVERY - fanonpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanonpedia.com' will expire on Mon 25 Mar 2024 16:05:42 GMT +0000. [18:06:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.23, 8.35, 7.14 [18:08:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.75, 7.07 [18:10:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.76, 7.14 [18:14:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 7.66, 7.27 [18:17:59] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [18:20:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.86, 7.49 [18:22:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.11, 7.25 [18:24:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [18:25:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.40, 8.38, 7.65 [18:27:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 7.90, 7.55 [18:28:26] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [18:57:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.47, 7.68, 7.30 [18:59:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.42, 7.25 [19:07:59] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:11:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.37, 7.16 [19:14:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.04, 7.35, 6.37 [19:15:32] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=mypediawiki (END - exit=256) [19:15:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:45] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.46, 6.62, 6.22 [19:17:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 7.91, 7.56 [19:18:15] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=mypediawiki (END - exit=256) [19:18:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.49, 8.46, 7.80 [19:22:44] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=mypediawiki (END - exit=0) [19:22:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.97, 7.29, 7.52 [19:24:34] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=beidipediawiki (END - exit=2) [19:24:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:52] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.88, 3.27, 2.60 [19:25:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.37, 7.97, 7.72 [19:26:49] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=beidipediawiki (END - exit=0) [19:26:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.22, 7.36, 6.62 [19:27:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.48, 7.56 [19:29:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.55, 6.58, 6.42 [19:30:44] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.74, 2.65, 2.57 [19:37:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 7.46, 7.29 [19:39:01] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.79, 2.82 [19:40:27] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.05, 3.61, 2.99 [19:41:01] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.78, 3.33, 2.76 [19:42:26] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.49, 3.19, 2.91 [19:46:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.71, 6.81, 6.38 [19:48:54] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.50, 6.55, 6.33 [20:03:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.56, 7.49, 7.88 [20:11:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.33, 8.18, 7.97 [20:13:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.73, 7.83 [20:33:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.66, 8.56, 7.68 [20:35:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 7.96, 7.56 [20:39:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.04, 8.34, 7.87 [20:41:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.24, 7.64, 7.67 [20:45:56] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.98, 2.49, 1.42 [20:47:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.68, 2.27, 1.48 [20:49:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 7.08, 7.18 [20:51:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.05, 7.16 [21:05:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.61, 8.25, 7.52 [21:07:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.62, 6.71, 6.04 [21:09:07] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.03, 6.11, 5.89 [21:09:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.82, 7.26, 7.31 [21:11:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 7.84, 7.51 [21:13:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 7.81, 7.54 [21:15:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.88, 7.58 [21:17:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.75, 7.56 [21:19:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.69, 8.32, 7.78 [21:23:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.97, 7.80 [21:33:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.01, 7.68, 6.70 [21:35:27] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.14, 7.46, 6.74 [21:37:23] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.40, 6.55, 6.51 [21:39:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.70, 7.64 [21:41:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.96, 7.42, 7.52 [22:01:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 6.81, 7.00 [22:03:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 7.10, 7.09 [22:05:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.77, 7.61, 7.26 [22:07:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 7.38, 7.21 [22:08:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.08, 7.05, 6.44 [22:10:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.37, 6.51, 6.32 [22:17:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.68, 6.18, 6.71 [22:21:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.16, 7.00 [22:29:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.39, 6.02, 6.57 [22:36:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.59, 7.25, 6.39 [22:38:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.84, 6.56, 6.22 [22:41:11] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:50:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.24, 6.85 [22:54:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.95, 6.56, 6.66 [23:09:11] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:13:22] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 6.37, 4.11, 2.64 [23:15:21] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.05, 3.73, 2.69 [23:17:21] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.20, 3.23, 2.63 [23:19:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.80, 6.77, 6.50 [23:21:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.53, 6.78, 6.55 [23:24:22] PROBLEM - swiftobject122 NTP time on swiftobject122 is WARNING: NTP WARNING: Offset 0.154250443 secs [23:30:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.09, 7.57, 6.44 [23:32:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.08, 7.01, 6.38 [23:35:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.07, 6.71 [23:36:14] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.18, 6.61, 6.41 [23:36:46] RECOVERY - swiftobject122 NTP time on swiftobject122 is OK: NTP OK: Offset -0.02289739251 secs [23:37:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.45, 6.89 [23:39:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.45, 7.16, 6.86 [23:41:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.03, 7.97, 7.17 [23:43:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.52, 7.09 [23:47:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.25, 6.52, 6.76 [23:50:52] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.53, 7.31, 6.58 [23:51:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 8.00, 7.34 [23:52:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.53, 7.54, 6.76 [23:53:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.33, 7.19, 7.14 [23:59:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.67, 7.52, 7.26