[00:01:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.01, 11.74, 11.05 [00:02:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.47, 4.31, 3.46 [00:03:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 11.93, 11.22 [00:04:06] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 2.57, 4.64, 2.81 [00:04:35] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.09, 3.87, 3.40 [00:05:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.79, 12.22, 11.43 [00:06:06] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.95, 3.41, 2.58 [00:06:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.98, 5.25, 2.89 [00:07:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 11.07, 11.08 [00:08:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.70, 2.49, 2.34 [00:08:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.77, 4.51, 2.90 [00:09:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.65, 11.69, 11.86 [00:10:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.99, 2.73, 3.13 [00:13:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 3.65, 2.92 [00:15:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 2.98, 2.76 [00:17:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.57, 9.13, 10.04 [00:23:02] PROBLEM - fid.koyomi-ya.net - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4066948890686035 seconds [00:24:08] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.05, 4.55, 3.51 [00:26:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 10.35, 10.25 [00:27:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.32, 9.27, 10.15 [00:28:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.21, 9.61, 9.99 [00:31:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.21, 3.69, 3.60 [00:32:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.52, 6.20, 4.32 [00:35:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.14, 2.97, 3.33 [00:36:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.68, 7.36, 5.28 [00:38:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.66, 6.11, 5.08 [00:42:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.87, 3.20, 3.24 [00:44:36] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.14, 3.38, 3.29 [00:46:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.30, 2.94, 3.14 [00:48:43] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:52:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.69, 3.76, 3.35 [00:52:35] RECOVERY - fid.koyomi-ya.net - reverse DNS on sslhost is OK: SSL OK - fid.koyomi-ya.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:56:18] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.19, 3.98, 3.58 [01:00:13] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.77, 3.09, 3.34 [01:20:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.14, 6.52 [01:22:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.32, 6.71, 6.43 [01:33:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.3806441128 secs [01:34:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.00, 4.84, 2.88 [01:36:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.11, 4.37, 2.97 [01:43:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.0269035399 secs [02:23:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.09, 6.53 [02:25:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.25, 6.63 [02:25:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.50, 3.14, 2.70 [02:26:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.84, 6.76, 6.53 [02:27:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.82, 4.16, 3.12 [02:29:08] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.25, 3.83, 3.13 [02:35:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.81, 4.11, 3.37 [02:36:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.17, 3.89, 3.40 [02:38:54] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.22, 3.20 [02:50:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:52:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 6.57, 6.38 [02:54:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.68, 6.20, 6.27 [03:00:52] RECOVERY - db112 Backups SQL on db112 is OK: FILE_AGE OK: /var/log/sql-backup.log is 49 seconds old and 363 bytes [03:01:05] RECOVERY - db112 Backups SQL on db112 is OK: FILE_AGE OK: /var/log/sql-backup.log is 61 seconds old and 363 bytes [03:04:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 10.49, 7.40, 4.82 [03:10:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.10, 3.71, 3.96 [03:18:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.07, 2.84, 3.38 [03:19:37] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:20:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 6.98, 6.34 [03:21:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.12, 3.10, 3.41 [03:22:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.06, 5.03, 2.89 [03:22:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.40, 6.40, 6.21 [03:23:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.65, 3.73, 3.60 [03:24:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.08, 5.32, 3.29 [03:27:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.66, 3.99, 3.76 [03:33:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.82, 3.00, 3.36 [03:50:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.08, 6.38 [03:52:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 6.99, 6.41 [03:54:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.29, 6.36, 6.25 [04:14:07] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.23, 3.42, 3.09 [04:16:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.98, 2.95, 2.96 [04:27:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1366934478 secs [04:33:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.07889515162 secs [04:50:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:14:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.02, 5.70, 3.02 [05:15:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.60, 2.95, 2.37 [05:17:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 2.59, 2.31 [05:18:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.08, 5.59, 3.78 [05:19:37] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:22:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.09, 6.59, 6.05 [05:24:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.69, 6.53, 6.09 [05:48:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.45, 3.15, 2.61 [05:50:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 2.38, 2.39 [05:53:25] PROBLEM - www.amusementparkarchives.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.amusementparkarchives.com could not be found [05:54:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.57, 3.29, 2.75 [05:56:47] PROBLEM - www.amusementparkarchives.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.amusementparkarchives.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:57:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.84, 3.21, 2.84 [06:28:07] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.51, 2.94, 1.67 [06:28:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.25, 3.89, 2.95 [06:30:06] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.15, 2.16, 1.54 [06:30:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.32, 3.05, 2.76 [06:37:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2211614251 secs [06:47:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.02018067241 secs [06:51:12] [02CreateWiki] 07Reception123 closed pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [06:51:15] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/CreateWiki/compare/8084dd88e10c...ce50bbbe131b [06:51:18] [02miraheze/CreateWiki] 07Universal-Omega 03ce50bbb - Support checking configs for states (#395) [06:51:33] [02CreateWiki] 07Reception123 closed pull request 03#396: Make WikiManager::create() more extendable - 13https://github.com/miraheze/CreateWiki/pull/396 [06:51:34] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/ce50bbbe131b...b408a4b9d50d [06:51:36] [02miraheze/CreateWiki] 07Universal-Omega 03b408a4b - Make WikiManager::create() more extendable (#396) [06:51:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.20, 6.76 [06:52:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.64, 5.01, 2.83 [06:53:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.21, 6.71, 6.62 [06:53:51] miraheze/CreateWiki - Reception123 the build has errored. [06:54:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.39, 5.70, 3.40 [07:22:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.35, 9.56, 8.81 [07:24:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.55, 8.64, 8.56 [07:36:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.10, 9.92, 8.94 [07:38:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.81, 9.53, 8.93 [08:22:49] PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864152 seconds old and 1753 bytes [09:01:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1153645515 secs [09:04:09] I just realized I never deployed the MirahezeMagic update to betaheze [09:06:33] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [09:07:54] !log [alex@test131] starting deploy of {'config': True, 'world': True, 'l10n': True} to all [09:07:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:33] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [09:13:02] ahh yes, 1 million depreciation warnings [09:13:19] *deprecation [09:22:07] !log [alex@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [09:22:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:22:19] damn [09:25:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.0859696269 secs [10:16:39] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:20:47] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:45:34] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:49:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:18:54] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:50:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:15:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:19:37] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:36:16] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.86, 5.78, 3.08 [12:36:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.38, 9.30, 8.33 [12:38:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.42, 8.82, 8.27 [12:40:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.65, 5.01, 3.46 [12:44:52] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:48:41] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:55:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.05, 2.96, 2.00 [12:57:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 2.52, 1.95 [13:03:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8542 MB (6% inode=99%); [13:10:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.80, 11.29, 9.83 [13:14:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.86, 11.41, 10.35 [13:18:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.24, 10.04, 10.00 [13:26:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.33, 10.29, 10.07 [13:35:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.90, 13.09, 10.91 [13:36:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.97, 12.81, 11.28 [13:39:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.27, 11.74, 10.89 [13:45:49] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 5.10, 3.64, 2.82 [13:46:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.16, 11.97, 11.62 [13:47:50] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.81, 3.50, 2.86 [13:47:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.46, 10.86, 10.59 [13:49:49] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.36, 3.78, 3.04 [13:49:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.57, 13.77, 12.32 [13:51:49] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.21, 3.57, 3.06 [13:51:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.95, 11.39, 10.90 [13:55:49] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.55, 3.24, 3.06 [14:03:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 11.18, 11.93 [14:05:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.94, 11.08, 10.75 [14:07:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 10.12, 10.43 [14:11:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.03, 9.09, 9.94 [14:11:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.72, 12.06, 11.87 [14:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.53, 11.45, 11.66 [14:15:56] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:18:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.83, 6.13, 3.44 [14:20:16] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.71, 4.48, 3.17 [14:23:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.84, 11.58, 11.48 [14:25:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.07, 11.02, 11.29 [14:35:38] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.03, 10.40, 9.49 [14:35:46] Orange_Star: hi. Still having trouble with graylog? it's pretty important to have it... [14:37:31] That made me remember, can you upload the fingerprint of graylog131 to [[Tech:SSH fingerprints]]? [14:37:31] https://meta.miraheze.org/wiki/Tech:SSH_fingerprints [14:37:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.33, 10.20, 9.50 [14:39:00] *graylog141, sorry [14:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 10.69, 10.77 [14:42:22] no, it's graylog131, sorry [14:43:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.01, 10.58, 10.72 [14:44:52] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:45:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.23, 10.09, 9.86 [14:49:07] sure [14:51:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.24, 10.28, 9.98 [14:55:32] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.29, 9.90, 9.92 [14:55:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.15, 9.23, 10.04 [14:57:04] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7967 MB (5% inode=99%); [15:00:47] Orange_Star: you don’t need to ssh into graylog [15:01:40] You can proxy from bast [15:04:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.70, 11.32, 10.56 [15:05:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 9.92, 9.71 [15:06:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.76, 11.07, 10.56 [15:08:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.08, 12.94, 11.30 [15:13:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.88, 9.70, 9.84 [15:14:35] welp I'm in graylog it seems [15:14:49] Reception123: It says I don't have permission to do anything [15:17:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.59, 11.54, 10.54 [15:18:27] I'll have a look [15:19:52] Orange_Star: try now? [15:20:10] it works [15:20:27] man I really do not like accessing these logs on a browser [15:21:12] Orange_Star: well it’s easier then having to search the logs separately on 6 different servers [15:21:26] yeah, before graylog it was quite bad to have look on each server [15:23:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 12.00, 11.20 [15:25:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.08, 12.15, 11.33 [15:29:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.66, 11.40, 11.28 [15:31:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.01, 10.78, 12.00 [15:33:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.42, 11.30, 11.20 [15:33:48] MacFan4000 or Reception123: There's some CentralAuth shenanigans with my betaheze account [15:33:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.38, 12.66, 12.54 [15:34:48] I don't have access to the global permissions, and Special:Preferences complains that "Your account has not been confirmed as belonging to the global account." [15:35:02] yeah it's because centralauth is separate on beta [15:35:14] you might need to create a new account [15:35:28] it's quite messy, we've always thought of reverting that and going back to the general centralauth as it's too messy [15:35:51] It's using the Betaheze account, my account was never copied from prod [15:35:52] usually invalidateUserSessions.php --wiki=betawiki--user= works for such issues [15:36:29] Special:MergeAccount says that I have like 30 local accounts at beta [15:36:41] I'll see what I can do [15:37:55] every time after an upgrade we talk about resetting beta but no one gets to it [15:38:22] the current situation isn't great though as my account (I use an alternate account anyway) always fails to log in and I need to invalidate my sessions every time [15:38:37] probably happened the same thing as in mine [15:39:05] Special:MergeAccount is useless, I'll try invalidateUserSessions [15:41:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.12, 11.57, 11.58 [15:42:56] !log [alex@test131] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=betawiki --user=OrangeStar (END - exit=0) [15:43:22] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 16.38, 10.77, 5.58 [15:43:45] yay it worked [15:43:47] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.55, 9.68, 11.66 [15:45:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.54, 10.13, 10.80 [15:45:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.871 second response time [15:45:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.32, 12.10, 12.32 [15:47:16] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.16, 7.41, 5.48 [15:47:49] Orange_Star: beta central auth is a bit messed up [15:47:56] just a lil bit [15:48:03] It needs its own memcache cluster Orange_Star [15:48:24] Reception123: the solution is segregating beta fully [15:49:14] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.53, 5.85, 5.12 [15:49:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.88, 11.58, 11.31 [15:51:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.52, 10.50, 11.75 [15:53:05] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.61, 7.75, 6.06 [15:53:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.54, 11.60, 11.33 [15:53:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.49, 12.42, 12.32 [15:55:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.72, 10.77, 11.07 [15:57:02] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.03, 6.65, 6.10 [15:57:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.74, 12.89, 11.79 [16:01:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.80, 11.48, 11.59 [16:03:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.31, 11.95, 11.72 [16:09:09] Just a reminder to those testing exts and skins on beta - a number of them aren’t quite up to date (https://github.com/miraheze/mediawiki/pulls?q=is%3Apr+is%3Aopen+base%3AREL1_40) so some errors that you may encounter may already have been fixed [16:09:44] I stopped trying to do updates due to the issues with deploying world [16:10:48] some extensions look impossible to test, like LdapAuthentication [16:10:57] how do you even test that? [16:12:42] It appears to be unmaintained in favor of LdapAuthentication2 [16:23:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.01, 5.18, 3.68 [16:25:08] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset 0.4746195376 secs [16:27:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.41, 3.83, 3.48 [16:29:12] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.65, 3.00, 3.21 [16:35:04] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8514 MB (6% inode=99%); [16:35:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.29, 2.66 [16:37:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.85, 2.57 [16:39:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 11.10, 11.90 [16:41:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.80, 11.08, 11.96 [16:41:51] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.79, 4.18, 3.62 [16:43:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.83, 3.32, 3.37 [16:44:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 17.23, 10.91, 6.42 [16:47:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.04, 11.93, 11.91 [16:48:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.42, 6.39, 5.66 [16:49:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.87, 11.28, 11.69 [16:51:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.67, 10.30, 10.62 [16:51:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.46, 12.02, 11.93 [16:52:06] Orange_Star: about testing CheckUser, I think we had a huge issue like a year ago about who can test it on Betaheze but ultimately, it should only be tested by an SRE-Steward [16:52:30] well good thing I'm an SRE [16:52:45] Oh you mean dual SRE-Steward [16:53:01] yeah [16:53:51] If it helps, I have signed the same NDA as you, and I checkusered myself, so who cares at the end of the day [16:54:09] yeah no, I understand that but Rhinos got in trouble a year ago for doing that on Betaheze [16:54:31] because CheckUser is governed by community policy and for SRE to do a CheckUser, it'd have to be under the ToU [16:54:48] I don't even know what to say [16:54:53] technicalities ¯\(ツ)/¯ [16:55:07] I'll make sure to resign after the upgrade is done then [16:55:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 10.99, 10.90 [16:56:03] splendid [16:56:07] 1984 [16:56:16] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.008684962988 secs [16:58:33] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [16:59:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.30, 11.34, 10.98 [17:02:33] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [17:04:05] [02mw-config] 07AgentIsai opened pull request 03#5207: Create global-renamer group on Meta - 13https://github.com/miraheze/mw-config/pull/5207 [17:04:31] Reception123: https://beta.betaheze.org/wiki/User_talk:Reception321#Hi_Reception! [17:05:02] miraheze/mw-config - AgentIsai the build passed. [17:11:49] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.97, 3.09, 2.80 [17:12:15] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.35, 7.07, 5.12 [17:12:35] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.55, 3.07 [17:14:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.87, 5.06, 4.62 [17:14:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.31, 3.55, 3.11 [17:15:49] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.37, 3.29, 2.96 [17:16:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.31, 3.96, 3.35 [17:19:49] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.47, 3.58, 3.17 [17:20:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 3.39, 3.28 [17:24:37] don't see any notif 😦 [17:24:43] really [17:25:01] well at least DiscussionTools work :) [17:25:14] try creating a topic at my talkpage [17:25:50] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.95, 3.32, 3.18 [17:31:49] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.01, 3.54, 3.30 [17:33:49] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.60, 3.17, 3.19 [17:39:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.204202354 secs [17:50:24] !log [reception@test131] starting deploy of {'world': True} to all [17:50:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:28] you know, other than the CentralAuth shenanigans and deploys not really working, I found surprisingly few bugs so far [17:53:37] !log [reception@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [17:53:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1217623651 secs [17:57:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.02397990227 secs [18:01:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07811719179 secs [18:02:05] [02MirahezeMagic] 07Universal-Omega opened pull request 03#418: RebuildVersionCache: remove usage of ExtensionProcessor - 13https://github.com/miraheze/MirahezeMagic/pull/418 [18:13:00] miraheze/MirahezeMagic - Universal-Omega the build passed. [18:29:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.57, 10.12, 11.62 [18:33:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.02, 10.54, 11.35 [18:35:05] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7967 MB (5% inode=99%); [18:35:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 10.57, 11.31 [18:35:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.53, 3.65, 2.94 [18:37:08] Agent: I have no problems merging your PR now, as apparently there's a technical reason for that. but are you sure the community shouldn't be consulted? [18:38:16] at least to establish a policy that that group is Steward only [18:39:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.50, 10.69, 11.11 [18:39:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.64, 3.10, 2.86 [18:40:11] PROBLEM - graylog131 NTP time on graylog131 is CRITICAL: NTP CRITICAL: Offset 0.8141998351 secs [18:42:15] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset 0.01277959347 secs [18:42:15] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.96, 5.18, 3.01 [18:43:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 11.49, 11.38 [18:44:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.84, 5.00, 3.24 [18:45:14] Orange_Star: beta CU is confusing [18:45:30] Because beta is a weird space [18:45:32] tell me about it [18:45:37] I think updates might be fine [18:45:50] But I think actual non ToU CU was banned [18:45:54] Ask Owen [18:48:32] was banned where? I know that I can't just use CU on the prod farm, but I didn't know that it also applied to beta [18:48:52] one thing's for sure, I ain't touching CU in beta again [18:53:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.20, 11.51, 11.23 [18:57:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 11.30, 11.24 [18:58:57] Orange_Star: re: global-renamer, because it'd just be a subset of Steward privs and required due to CentralAuth inefficiencies, and because it wouldn't be an electable position/granted to non-stewards, a community consultation isn't required [19:03:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 10.73, 11.86 [19:05:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.65, 11.93, 12.18 [19:07:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.78, 12.17, 11.64 [19:13:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 11.62, 11.63 [19:21:50] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.233802408 secs [19:23:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 3.06, 2.33 [19:25:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 2.60, 2.25 [19:27:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.10, 12.06, 11.53 [19:41:50] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.008857578039 secs [19:50:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.50, 6.82, 5.79 [19:54:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.19, 3.94, 2.85 [19:58:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.01, 3.85, 3.08 [20:00:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.23, 2.93, 2.84 [20:06:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ad2f0279ac14 [20:06:24] [02miraheze/puppet] 07paladox 03ad2f027 - swift::storage: increase max connections for rsync to 5 * 4 [20:06:25] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [20:06:30] [02puppet] 07paladox opened pull request 03#3205: swift::storage: increase max connections for rsync to 5 * 4 - 13https://github.com/miraheze/puppet/pull/3205 [20:06:41] [02puppet] 07paladox closed pull request 03#3205: swift::storage: increase max connections for rsync to 5 * 4 - 13https://github.com/miraheze/puppet/pull/3205 [20:06:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5bf52439c10b...b8deb27dcd60 [20:06:46] [02miraheze/puppet] 07paladox 03b8deb27 - swift::storage: increase max connections for rsync to 5 * 4 (#3205) [20:06:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [20:06:51] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [20:07:54] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.87, 6.32, 5.29 [20:08:21] [02CreateWiki] 07Universal-Omega opened pull request 03#399: Remove extra whitespace - 13https://github.com/miraheze/CreateWiki/pull/399 [20:09:51] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.14, 5.57, 5.14 [20:09:52] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b408a4b9d50d...f78899489775 [20:09:53] [02miraheze/CreateWiki] 07Universal-Omega 03f788994 - Remove extra whitespace (#399) [20:09:54] [02CreateWiki] 07paladox closed pull request 03#399: Remove extra whitespace - 13https://github.com/miraheze/CreateWiki/pull/399 [20:15:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.45, 10.40, 11.69 [20:16:49] miraheze/CreateWiki - Universal-Omega the build passed. [20:21:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.65, 9.64, 11.51 [20:21:50] miraheze/CreateWiki - paladox the build passed. [20:25:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.06, 10.49, 11.02 [20:25:42] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:27:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.65, 11.27, 11.27 [20:27:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.27, 10.94, 11.43 [20:28:15] PROBLEM - swiftobject121 PowerDNS Recursor on swiftobject121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:31:09] PROBLEM - swiftobject121 SSH on swiftobject121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.60, 10.68, 11.34 [20:35:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.58, 12.06, 11.36 [20:35:17] RECOVERY - swiftobject121 SSH on swiftobject121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [20:35:23] RECOVERY - swiftobject121 PowerDNS Recursor on swiftobject121 is OK: DNS OK: 1.959 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [20:35:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 11.68, 11.64 [20:36:38] RECOVERY - swiftobject121 APT on swiftobject121 is OK: APT OK: 15 packages available for upgrade (0 critical updates). [20:37:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.75, 10.98, 11.42 [20:39:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.01, 11.81, 11.64 [20:41:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.01, 11.36, 11.41 [20:41:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 11.72, 11.63 [20:45:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.35, 11.61, 11.44 [20:47:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 10.33, 10.97 [20:51:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.87, 8.52, 10.14 [20:53:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1493034661 secs [20:56:54] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:58:06] PROBLEM - swiftobject121 PowerDNS Recursor on swiftobject121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:02:46] RECOVERY - swiftobject121 APT on swiftobject121 is OK: APT OK: 15 packages available for upgrade (0 critical updates). [21:03:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.98, 11.65, 11.04 [21:04:26]