[00:02:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.88, 4.11, 3.23 [00:02:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:03:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.41, 5.38, 3.70 [00:04:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 12.44, 10.29, 4.98 [00:04:23] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.56, 11.47, 9.53 [00:04:25] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:06:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.65, 3.87, 3.33 [00:06:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.92, 10.01, 9.24 [00:07:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.05, 3.97, 3.52 [00:10:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.28, 3.19, 3.18 [00:10:13] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.01, 3.80, 3.71 [00:11:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.63, 2.98, 3.23 [00:12:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.45, 7.65, 4.71 [00:12:12] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.97, 2.82, 3.36 [00:12:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.24, 10.35, 9.83 [00:14:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.12, 6.30, 4.58 [00:14:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.70, 9.66, 9.63 [00:16:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.86, 3.94, 3.51 [00:19:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 2.68, 3.11 [00:26:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.99, 7.79, 5.68 [00:31:20] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 30848 MB (23% inode=98%); [00:32:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.38, 7.94, 6.61 [00:36:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.94, 3.38 [00:38:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.74, 8.66, 7.19 [00:38:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.71, 4.02, 3.48 [00:40:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.39, 7.90, 7.12 [00:42:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 3.32, 3.35 [00:44:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.13, 8.35, 7.39 [00:44:38] [02mw-config] 07thecutefinder commented on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501008419 [00:45:09] [02mw-config] 07thecutefinder commented on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501008482 [00:46:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.25, 7.07, 7.03 [00:48:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.22, 7.63, 7.22 [00:49:42] [02mw-config] 07thecutefinder edited a comment on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501008419 [00:50:04] [02mw-config] 07thecutefinder edited a comment on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501008419 [00:50:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.55, 7.26, 7.15 [00:52:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.73, 9.35, 7.92 [00:56:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.64, 7.58, 7.63 [01:01:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.43, 9.97, 9.22 [01:02:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.75, 4.04, 2.21 [01:02:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 17.14, 9.42, 8.10 [01:02:34] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:04:02] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.76, 3.87, 2.40 [01:04:39] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 5.02, 3.68, 3.00 [01:06:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.79, 3.20, 2.33 [01:06:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.72, 10.79, 9.19 [01:07:28] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [01:08:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.39, 7.57, 7.81 [01:08:29] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.10, 3.52, 3.11 [01:08:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.38, 3.81, 3.32 [01:09:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.89, 12.20, 10.40 [01:10:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.56, 7.92, 7.90 [01:10:26] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.77, 3.28, 3.06 [01:10:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.71, 3.97, 3.45 [01:14:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.55, 7.54, 7.81 [01:14:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 3.26, 3.28 [01:16:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.39, 10.96, 10.43 [01:16:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.61, 7.82, 7.87 [01:17:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.39, 11.70, 11.09 [01:18:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.54, 7.42, 7.70 [01:18:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.63, 3.83, 3.53 [01:22:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.18, 3.90, 3.61 [01:26:08] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.87, 5.49, 6.80 [01:27:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.57, 9.41, 10.01 [01:28:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.23, 3.70, 3.71 [01:30:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.12, 6.82, 7.11 [01:32:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 2.94, 3.40 [01:32:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.27, 9.37, 10.12 [01:32:53] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.50, 3.25, 3.00 [01:34:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.50, 11.55, 10.61 [01:34:49] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.33, 2.92, 2.91 [01:36:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.13, 3.32, 3.47 [01:36:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.24, 7.63, 7.27 [01:36:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.86, 10.29, 10.28 [01:36:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.75, 10.46, 10.41 [01:38:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.48, 2.91, 3.30 [01:38:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.18, 10.91, 10.57 [01:42:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.60, 6.60, 7.02 [01:42:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.83, 11.63, 10.97 [01:46:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.27, 9.47, 10.02 [01:48:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.96, 8.83, 9.94 [01:50:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.00, 8.53, 7.68 [01:50:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.33, 3.21 [01:52:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.70, 3.01, 3.09 [01:56:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 1.40, 6.80, 7.60 [01:56:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 74.208.203.152/cpweb, 2607:f1c0:1800:26f::1/cpweb [01:56:20] PROBLEM - cp33 SSH on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 134.06, 67.54, 28.29 [01:56:28] PROBLEM - cp33 HTTPS on cp33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.05, 5.48, 3.96 [01:57:18] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [01:57:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 74.208.203.152/cpweb, 2607:f1c0:1800:26f::1/cpweb [01:58:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.29, 4.91, 6.80 [01:58:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:58:15] RECOVERY - cp33 SSH on cp33 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:58:25] RECOVERY - cp33 HTTPS on cp33 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3714 bytes in 0.528 second response time [01:59:16] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 3% [01:59:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:01:49] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.35, 5.00, 2.52 [02:02:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.26, 9.71, 9.34 [02:02:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:03:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.04, 5.84, 4.21 [02:04:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.13, 12.10, 10.25 [02:06:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 11.09, 10.11 [02:07:21] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [02:07:47] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.62, 3.81, 3.15 [02:07:59] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.18, 6.17, 4.73 [02:09:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.67, 2.80, 2.86 [02:09:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.90, 9.67, 9.73 [02:11:36] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.01, 11.11, 10.01 [02:17:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 11.08, 10.45 [02:17:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.23, 10.63, 10.06 [02:18:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.92, 3.13, 3.91 [02:19:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.60, 9.61, 9.77 [02:21:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.15, 3.25, 3.74 [02:23:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.55, 9.56, 10.00 [02:23:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 3.67, 3.85 [02:29:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.63, 2.46, 3.27 [02:36:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.52, 11.70, 10.28 [02:39:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.18, 11.46, 10.50 [02:39:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.61, 1.77, 3.94 [02:40:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 11.27, 10.47 [02:40:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.33, 3.99, 3.65 [02:40:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.68, 11.63, 10.69 [02:44:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.37, 9.36, 9.86 [02:44:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.85, 3.83, 3.67 [02:45:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 1.83, 3.26 [02:48:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 3.04, 3.38 [03:01:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 10.13, 9.71 [03:02:32] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:03:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.59, 4.40, 3.66 [03:03:48] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.81, 11.09, 10.09 [03:04:30] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [03:05:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.52, 11.29, 10.30 [03:07:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.81, 9.67, 9.83 [03:10:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.78, 12.23, 11.06 [03:14:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 11.20, 10.98 [03:14:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.68, 3.62, 3.79 [03:16:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 3.75, 3.81 [03:18:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.35, 6.41, 3.95 [03:18:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.99, 3.52, 3.72 [03:18:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.67, 11.32, 11.04 [03:20:38] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.18, 5.84, 4.06 [03:24:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.25, 2.68, 3.28 [03:24:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.78, 11.15, 11.13 [03:28:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.95, 11.31, 11.11 [03:30:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 10.61, 10.88 [03:33:59] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:39:17] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [03:43:18] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [03:49:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.41, 10.58, 9.90 [03:51:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.47, 10.05, 9.79 [03:52:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.34, 9.08, 10.00 [04:03:41] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [04:05:41] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1322215199 secs [04:07:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.49, 3.58, 3.07 [04:09:41] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09295770526 secs [04:09:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.36, 3.05 [04:11:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.55, 3.06, 2.97 [04:17:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.34, 3.45, 3.20 [04:19:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 3.72, 3.33 [04:21:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.48, 3.30, 3.22 [04:33:24] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:47:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.44, 3.04 [04:49:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.87, 3.40, 3.08 [04:53:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.66, 4.26, 3.54 [04:55:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 3.22, 3.24 [05:03:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 41.66, 21.68, 9.51 [05:23:39] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.05, 2.28, 3.98 [05:27:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.08, 1.69, 3.34 [05:28:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.41, 3.92, 2.17 [05:30:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.60, 2.96, 2.03 [05:39:41] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.2044523954 secs [05:49:42] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.04657527804 secs [06:08:44] PROBLEM - cloud12 Current Load on cloud12 is CRITICAL: CRITICAL - load average: 63.65, 36.24, 22.98 [06:08:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.54, 11.94, 9.67 [06:10:56] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 14.53, 9.90, 7.03 [06:11:03] PROBLEM - mw131 Disk Space on mw131 is CRITICAL: DISK CRITICAL - free space: / 1388 MB (5% inode=82%); [06:11:13] PROBLEM - mw121 Disk Space on mw121 is CRITICAL: DISK CRITICAL - free space: / 477 MB (2% inode=81%); [06:12:44] PROBLEM - cloud12 Current Load on cloud12 is WARNING: WARNING - load average: 38.52, 45.37, 30.36 [06:12:56] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.66, 8.80, 6.98 [06:14:45] RECOVERY - cloud12 Current Load on cloud12 is OK: OK - load average: 20.46, 36.59, 28.98 [06:16:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 11.82, 10.81 [06:20:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.06, 6.30, 4.11 [06:22:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.53, 4.82, 3.83 [06:24:14] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1485957503 secs [06:24:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.47, 10.71, 10.53 [06:26:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 10.94, 10.67 [06:30:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 9.25, 10.05 [06:42:09] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.93, 2.83, 2.25 [06:44:09] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.52, 2.74, 2.29 [06:46:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09977737069 secs [06:59:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.23, 9.69, 8.77 [07:01:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.45, 9.65, 8.87 [08:11:12] RECOVERY - mw121 Disk Space on mw121 is OK: DISK OK - free space: / 6052 MB (27% inode=81%); [08:13:03] RECOVERY - mw131 Disk Space on mw131 is OK: DISK OK - free space: / 9744 MB (41% inode=82%); [08:37:58] PROBLEM - db101 Backups SQL on db101 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209604 seconds old and 1879 bytes [08:41:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.63, 10.03, 8.69 [08:43:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.53, 10.39, 9.01 [08:45:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.84, 9.82, 8.97 [09:21:47] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:30:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.60, 3.82, 2.54 [09:32:52] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.51, 2.94, 2.36 [09:40:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.73, 11.01, 8.76 [09:42:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 10.35, 8.81 [09:44:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.19, 9.90, 8.83 [09:50:35] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:42:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.62, 9.07, 7.85 [10:44:43] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_landing] [10:44:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.06, 8.44, 7.79 [11:10:40] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:22:18] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1642632484 secs [11:28:18] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05668112636 secs [11:35:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.64, 9.84, 8.61 [11:37:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.47, 9.47, 8.62 [11:59:13] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.34, 2.85, 1.78 [12:01:00] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:02:57] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [12:03:12] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.59, 2.97, 2.09 [12:16:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.55, 10.26, 9.75 [12:18:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.61, 9.27, 9.45 [12:21:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.28, 10.13, 9.74 [12:25:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.31, 9.59, 9.65 [12:39:12] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:40:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.52, 10.55, 9.85 [12:42:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.64, 9.86, 9.69 [13:10:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.56, 5.79, 3.07 [13:12:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.10, 6.22, 3.59 [13:19:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.96, 5.29, 4.26 [13:21:07] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.18, 5.01, 4.30 [13:35:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.18, 11.81, 10.08 [13:35:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.32, 11.23, 9.24 [13:37:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.92, 11.20, 10.05 [13:37:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 10.55, 9.21 [13:38:37] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:39:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.84, 11.78, 10.40 [13:39:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.95, 11.98, 9.88 [13:40:26] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.344691664 secs [13:41:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 11.31, 9.89 [13:43:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.70, 11.66, 10.75 [13:43:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.73, 9.79, 9.52 [13:47:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.80, 11.89, 11.04 [13:49:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.31, 11.07, 10.85 [13:49:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.37, 10.65, 9.99 [13:51:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.24, 9.53, 9.67 [13:56:26] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02264437079 secs [13:58:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.66, 3.49, 2.45 [14:02:39] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 3.11, 2.58 [14:05:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.44, 11.93, 11.23 [14:06:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.56, 11.22, 11.04 [14:20:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.48, 9.25, 10.10 [14:26:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.73, 9.84, 10.00 [14:34:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.25, 9.21, 9.86 [14:40:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.95, 11.64, 10.55 [14:42:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.33, 11.10, 10.49 [14:48:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.13, 10.90, 10.52 [14:48:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 10.06, 9.79 [14:50:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.45, 11.05, 10.63 [14:50:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.91, 10.97, 10.14 [14:52:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.02, 9.99, 9.87 [14:58:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.20, 11.09, 10.75 [15:00:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.55, 10.59, 10.62 [15:08:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.11, 9.56, 10.18 [15:23:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.12, 3.48, 2.66 [15:25:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 2.97, 2.57 [15:33:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.33, 11.78, 10.23 [15:34:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 10.40, 10.11 [15:35:52] [02mw-config] 07redbluegreenhat commented on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501155176 [15:36:49] [02mw-config] 07redbluegreenhat commented on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501155421 [15:36:50] [02mw-config] 07redbluegreenhat closed pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188 [15:36:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 10.97, 10.35 [15:37:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 11.14, 10.41 [15:38:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 10.68, 10.33 [15:39:21] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:40:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.86, 11.09, 10.49 [15:41:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.84, 11.80, 10.81 [15:41:46] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:42:51] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.79, 7.43, 4.49 [15:42:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.16, 10.54, 10.38 [15:43:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 10.24, 10.35 [15:44:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.99, 9.82, 10.13 [15:45:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.31, 9.26, 9.98 [15:46:44] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.18, 7.00, 5.07 [15:48:40] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.31, 4.95, 4.55 [15:49:45] [02mw-config] 07redbluegreenhat reviewed pull request 03#5187 commit - 13https://github.com/miraheze/mw-config/pull/5187#discussion_r1161305867 [15:51:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.30, 9.83, 9.91 [15:53:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.60, 10.77, 10.22 [15:55:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.21, 9.96, 10.02 [15:55:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.67, 11.25, 10.30 [16:00:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.15, 10.36, 10.10 [16:03:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 11.63, 11.09 [16:04:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.39, 12.09, 10.79 [16:05:34] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.24, 13.10, 11.67 [16:06:48] [02mw-config] 07thecutefinder commented on pull request 03#5181: Update genshinimpactwiki settings again - 13https://github.com/miraheze/mw-config/pull/5181#issuecomment-1501161428 [16:07:06] [02mw-config] 07thecutefinder commented on pull request 03#5188: Add $wgCleanSignatures = false - 13https://github.com/miraheze/mw-config/pull/5188#issuecomment-1501161489 [16:09:03] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:09:46] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:14:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.57, 10.50, 10.91 [16:16:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 11.27, 11.14 [16:17:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.47, 11.57, 11.88 [16:18:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.67, 11.23, 11.13 [16:19:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.26, 12.48, 12.17 [16:24:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.87, 12.54, 11.65 [16:26:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.37, 11.69, 11.45 [16:31:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.15, 11.46, 11.88 [16:32:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.40, 11.93, 11.53 [16:35:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 11.71, 11.87 [16:38:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.93, 10.79, 11.48 [16:44:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 11.31, 11.87 [16:48:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.74, 11.31, 11.68 [16:50:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.70, 11.43, 11.68 [16:58:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.33, 4.25, 3.10 [16:58:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.33, 8.95, 10.04 [17:00:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.64, 8.19, 4.70 [17:00:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.30, 3.95, 3.13 [17:02:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.28, 9.78, 10.17 [17:04:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.66, 6.72, 4.98 [17:04:30] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.84, 2.85, 2.86 [17:04:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.97, 11.81, 10.87 [17:06:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 9.23, 10.14 [17:06:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 11.24, 10.80 [17:08:09] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.71, 2.84, 2.19 [17:10:09] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.02, 2.91, 2.30 [17:12:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.57, 9.90, 10.23 [17:12:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.82, 9.08, 10.12 [17:14:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.45, 9.20, 9.94 [17:18:31] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.4310245216 secs [17:28:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.42, 3.20, 2.01 [17:30:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.58, 3.01, 2.09 [17:34:32] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.005850017071 secs [17:36:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.04, 10.93, 10.09 [17:36:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.81, 10.52, 10.03 [17:38:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.91, 11.12, 10.28 [17:38:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.31, 10.26, 10.00 [17:40:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.42, 9.94, 9.91 [17:48:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.32, 11.95, 11.00 [17:51:51] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 3.66, 2.84 [17:52:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.75, 10.91, 10.33 [17:53:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.47, 3.31, 2.82 [18:00:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.67, 9.55, 10.03 [18:02:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 11.12, 11.58 [18:10:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.55, 10.83, 11.04 [18:12:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.37, 11.04, 11.10 [18:18:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.38, 8.90, 10.17 [18:28:12] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [18:30:26] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1557065547 secs [18:36:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.84, 9.95, 9.95 [18:38:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.91, 9.32, 9.73 [18:44:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.72, 3.51, 3.01 [18:46:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.82, 3.27, 2.97 [18:47:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.99, 10.36, 10.05 [18:49:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.88, 12.61, 10.93 [18:50:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.90, 10.38, 8.92 [18:52:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.15, 3.69, 3.29 [18:52:26] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.05304762721 secs [18:54:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.42, 3.29, 3.20 [18:56:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.55, 11.29, 9.81 [19:05:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.23, 11.31, 11.89 [19:08:59] !log [salt-user@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=65280) [19:09:01] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=256) [19:09:02] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=256) [19:09:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:13] !log [salt-user@mw131] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=256) [19:09:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.29, 11.72, 11.93 [19:09:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:09:52] !log [salt-user@mw132] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=0) [19:09:53] !log [salt-user@mw142] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=0) [19:09:54] !log [salt-user@mw141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=projectsekaiwiki (END - exit=0) [19:10:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:01] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [19:21:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 11.06, 11.60 [19:26:23] RECOVERY - gs.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - gs.sidem.wiki reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:28:26] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1132374704 secs [19:29:00] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.65, 5.76, 4.20 [19:30:56] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.54, 6.97, 4.85 [19:32:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.97, 9.82, 11.42 [19:32:52] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.36, 7.01, 5.11 [19:33:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.74, 10.99, 11.01 [19:34:49] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.16, 5.31, 4.72 [19:37:10] !log [salt-user@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=65280) [19:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.19, 11.65, 11.32 [19:37:28] !log [salt-user@mw141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=0) [19:37:31] !log [salt-user@mw142] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=0) [19:37:32] !log [salt-user@mw132] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=0) [19:37:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:34] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=0) [19:37:35] !log [salt-user@mw131] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=0) [19:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:36] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=bluepageswiki (END - exit=0) [19:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:38:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:43] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:42:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.36, 12.91, 12.01 [19:44:26] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.06528365612 secs [19:45:06]