[00:02:00] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.41, 4.58, 3.90 [00:02:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.68, 4.88, 3.88 [00:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 6.97, 6.85 [00:04:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.53, 7.27, 6.96 [00:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 7.78, 7.20 [00:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.80, 8.42, 7.57 [00:11:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.68, 3.51, 3.77 [00:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.76, 7.42 [00:15:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.97, 3.68, 3.95 [00:18:38] [02.github] 07The-Voidwalker commented on pull request 03#31: Add PHP 8.2 tests - 13https://github.com/miraheze/.github/pull/31#issuecomment-1574480383 [00:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.58, 7.66, 7.42 [00:19:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.32, 3.37, 3.74 [00:21:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.58, 3.08, 3.59 [00:23:11] [02.github] 07Universal-Omega commented on pull request 03#31: Add PHP 8.2 tests - 13https://github.com/miraheze/.github/pull/31#issuecomment-1574484099 [00:23:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 2.75, 3.40 [00:23:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.08, 4.12, 3.76 [00:25:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.12, 3.77, 3.68 [00:25:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.56, 7.03, 6.30 [00:27:45] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.96, 6.36, 6.15 [00:28:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.29, 7.43, 7.68 [00:30:24] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [00:32:07] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.91, 3.23, 2.39 [00:32:22] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [00:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.93, 7.91, 7.79 [00:33:20] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.23, 7.27, 4.88 [00:33:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 2.78, 3.26 [00:34:07] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.97, 3.17, 2.48 [00:34:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 7.70, 7.72 [00:35:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.66, 4.85, 3.77 [00:37:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:37:11] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.81, 7.14, 5.46 [00:39:06] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.84, 5.94, 5.22 [00:43:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.27, 3.82, 3.83 [00:44:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.11, 7.33 [00:44:55] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.82, 6.43, 5.63 [00:45:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.58, 4.21, 3.95 [00:46:50] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.17, 5.88, 5.52 [00:50:43] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.03, 7.69, 6.36 [00:51:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.88, 2.92, 3.46 [00:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.99, 7.80 [00:53:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.33, 2.69, 3.30 [00:56:02] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.96, 3.66, 3.10 [00:56:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.69, 7.76, 6.94 [00:57:57] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.15, 2.82, 2.87 [00:58:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.83, 5.93, 6.37 [01:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 7.13, 7.39 [01:04:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 10.26, 11.60 [01:06:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.95, 7.41, 6.67 [01:07:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:10:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.32, 11.71, 11.64 [01:10:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.85, 7.00, 6.76 [01:11:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.43, 3.66, 3.03 [01:12:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.90, 4.83, 6.00 [01:13:18] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [01:13:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.57, 3.12, 2.90 [01:13:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.64, 3.60, 3.31 [01:15:18] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.027 second response time [01:15:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 3.56, 3.32 [01:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.55, 7.86 [01:17:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.52, 3.40, 3.28 [01:18:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.66, 7.85 [01:19:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.71, 3.04 [01:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.97, 7.95 [01:26:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 8.16, 8.04 [01:31:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.80, 4.60, 3.74 [01:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.44, 7.62, 7.96 [01:43:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.56, 3.66, 3.86 [01:45:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.49, 4.36, 4.07 [01:45:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.65, 3.37, 2.66 [01:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.83, 7.87, 7.93 [01:47:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.15, 3.74, 3.87 [01:47:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.93, 3.22, 2.70 [01:52:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.79, 7.23, 6.35 [01:54:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.11, 6.65, 6.24 [01:59:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.11, 3.92, 3.73 [02:06:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 3.67, 3.03 [02:08:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.47, 3.02 [02:09:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.27, 3.78, 3.88 [02:11:16] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1616089642 secs [02:12:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.64, 3.17 [02:14:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.94, 3.47, 3.17 [02:15:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.15, 2.89, 3.40 [02:16:14] PROBLEM - wiki.mcsoft.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mcsoft.org All nameservers failed to answer the query. [02:19:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.59, 3.17, 3.14 [02:20:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.43, 6.79, 6.46 [02:22:42] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.57, 6.06, 6.24 [02:24:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.93, 3.91, 3.62 [02:25:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.17, 3.74, 3.40 [02:27:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.53, 2.92, 3.13 [02:28:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.96, 3.22, 3.43 [02:29:16] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05652600527 secs [02:29:42] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [02:29:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 2.78, 3.25 [02:31:41] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.054 second response time [02:31:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.39, 3.66, 3.50 [02:33:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.38, 2.81, 3.20 [02:37:32] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [02:39:26] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [02:45:19] PROBLEM - wiki.mcsoft.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404912948608398 seconds [02:48:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.19, 7.33, 7.89 [02:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.91, 8.83, 8.37 [02:55:33] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [03:01:18] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.063 second response time [03:02:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 13.43, 7.08, 4.31 [03:02:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.2030189633 secs [03:09:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.71, 3.29, 2.73 [03:10:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 7.25, 7.92 [03:11:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.95, 3.10, 2.73 [03:11:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.97, 3.35, 3.69 [03:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.44, 7.89 [03:14:19] RECOVERY - wiki.mcsoft.org - reverse DNS on sslhost is OK: SSL OK - wiki.mcsoft.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:14:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.19, 5.22, 2.73 [03:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 7.69, 7.95 [03:15:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.36, 2.74, 3.36 [03:16:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.33, 5.57, 3.19 [03:16:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.56, 7.83, 7.95 [03:20:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.80, 3.08 [03:22:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.66, 3.35, 2.98 [03:24:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.39, 3.84, 3.19 [03:24:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.01955065131 secs [03:27:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.76, 3.65, 3.26 [03:29:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.29, 3.13, 3.11 [03:35:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.52, 4.32, 3.58 [03:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 7.55, 7.98 [03:37:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.16, 3.90, 3.52 [03:41:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 3.03, 3.27 [03:42:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.16, 8.17, 8.06 [03:58:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.74, 3.41, 3.13 [04:00:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.37, 2.97, 2.99 [04:07:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:12:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 7.77, 7.98 [04:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.84, 7.98 [04:16:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 7.88, 7.97 [04:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.11, 8.86, 8.33 [04:24:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.58, 5.79, 3.16 [04:26:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.85, 6.08, 3.61 [04:28:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.96, 6.68, 6.15 [04:28:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.46, 3.43, 2.94 [04:30:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.86, 5.93, 5.94 [04:32:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.37, 3.76, 3.22 [04:34:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.54, 7.95 [04:36:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.80, 4.43, 3.58 [04:37:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:40:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.12, 3.40, 3.37 [04:40:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.88, 7.97 [04:57:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.18, 6.66, 6.21 [04:57:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.40, 4.03, 3.51 [04:59:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.78, 3.89, 3.52 [05:02:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.48, 6.58, 6.27 [05:03:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.27, 4.10, 3.68 [05:04:01] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:06:00] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.144 second response time [05:08:05] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:09:58] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.030 second response time [05:11:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.01, 3.86, 3.93 [05:19:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.15, 2.40, 3.25 [05:24:24] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [05:25:45] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.89, 2.98 [05:25:59] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:27:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 3.09, 2.80 [05:28:24] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.026 second response time [05:31:16] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1644231677 secs [05:38:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.99, 5.56, 3.40 [05:40:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.37, 4.23, 3.18 [05:41:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.81, 3.28, 2.68 [05:43:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.92, 3.15, 2.71 [05:47:30] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [05:49:29] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.054 second response time [05:50:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.51, 6.96, 6.13 [05:52:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.23, 6.26, 5.98 [05:53:16] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07080295682 secs [05:55:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:04:24] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:07:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:09:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.58, 3.31, 2.66 [06:11:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.18, 3.30, 2.58 [06:13:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.40, 2.70, 2.45 [06:25:44] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:29:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.19, 3.30, 2.61 [06:31:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.89, 2.68, 2.46 [06:33:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.99, 3.94 [06:34:08] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:37:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:38:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.87, 6.02, 3.41 [06:39:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.23, 4.04, 3.95 [06:40:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.21, 5.06, 3.39 [06:51:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.77, 3.96 [06:51:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.73, 7.10, 6.27 [06:53:18] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.55, 6.92, 6.31 [06:53:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.26, 4.04, 4.04 [06:55:15] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.42, 6.30, 6.15 [06:55:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:07:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.79, 3.59, 3.90 [07:14:24] [02.github] 07RhinosF1 commented on pull request 03#31: Add PHP 8.2 tests - 13https://github.com/miraheze/.github/pull/31#issuecomment-1574741339 [07:17:02] [02.github] 07RhinosF1 commented on pull request 03#31: Add PHP 8.2 tests - 13https://github.com/miraheze/.github/pull/31#issuecomment-1574741832 [07:20:28] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [07:21:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.12, 10.74, 11.87 [07:21:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.46, 2.83, 3.29 [07:22:27] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.270 second response time [07:25:28] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:26:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 10.91, 11.69 [07:31:26] [02.github] 07Universal-Omega commented on pull request 03#31: Add PHP 8.2 tests - 13https://github.com/miraheze/.github/pull/31#issuecomment-1574747655 [07:35:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.64, 11.51, 11.50 [07:36:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.37, 8.45, 10.10 [07:37:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.75, 11.48, 11.50 [07:38:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 6.26, 6.02 [07:39:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.67, 11.89, 11.62 [07:40:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.35, 10.71, 10.55 [07:41:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.78, 11.65, 11.56 [07:42:51] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.56, 6.66, 6.28 [07:43:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.80, 11.94, 11.66 [07:50:35] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.00, 7.58, 6.71 [07:51:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.14, 11.70, 11.72 [07:52:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.71, 10.83, 11.04 [07:52:32] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.53, 6.71, 6.49 [07:56:50] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [07:57:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.83, 12.38, 11.97 [07:58:46] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.098 second response time [07:59:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.84, 10.87, 11.49 [08:02:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.75, 10.83 [08:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.91, 10.82, 11.24 [08:04:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.54, 10.97, 10.89 [08:07:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 12.00, 11.59, 11.48 [08:12:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.47, 12.36, 11.45 [08:14:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 11.40, 11.21 [08:15:12] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [08:18:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.32, 11.41, 11.18 [08:19:03] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.040 second response time [08:19:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.28, 10.60, 10.60 [08:20:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.37, 11.04, 11.08 [08:21:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.76, 10.30 [08:21:57] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [08:23:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.91, 9.55, 10.13 [08:23:55] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.031 second response time [08:27:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.76, 11.05, 10.62 [08:29:05] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.89, 3.25, 2.03 [08:31:03] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.56, 2.98, 2.07 [08:32:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.50, 8.98, 10.03 [08:35:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.86, 11.05, 10.57 [08:36:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.36, 10.79, 10.56 [08:37:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.27, 10.87, 10.58 [08:38:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.65, 11.05, 10.66 [08:40:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.12, 10.84, 10.62 [08:42:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 11.34, 10.83 [08:46:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 10.75, 10.72 [08:46:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.60, 6.83, 6.39 [08:46:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1238734722 secs [08:48:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.15, 6.48, 6.31 [08:53:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.13, 9.39, 10.04 [08:54:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.07989147305 secs [09:05:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.36, 12.37, 10.83 [09:06:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.98, 11.53, 10.75 [09:07:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.33, 11.55, 10.70 [09:08:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.44, 10.93, 10.62 [09:15:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 11.31, 10.79 [09:19:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.59, 11.57, 11.03 [09:22:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.47, 10.83, 10.55 [09:23:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.70, 12.69, 11.59 [09:24:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.50, 10.04, 10.31 [09:25:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.23, 11.42, 11.26 [09:26:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.71, 9.04, 9.93 [09:33:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.39, 11.35, 11.04 [09:35:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 11.35, 11.10 [09:39:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.10, 11.00, 10.92 [09:41:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.67, 10.59, 10.78 [09:42:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.02, 12.45, 10.78 [09:43:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.47, 11.88, 11.24 [09:44:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.06, 11.61, 10.69 [09:45:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 10.96, 10.98 [09:48:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.42, 11.45, 10.76 [09:50:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.27, 10.95, 10.66 [09:53:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.31, 10.79, 10.83 [09:55:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.16, 10.41, 10.68 [10:02:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.16, 10.77, 10.57 [10:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.50, 11.39, 10.95 [10:05:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.81, 11.90, 11.22 [10:07:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.05, 12.62, 11.59 [10:11:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.18, 11.97, 11.57 [10:15:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.20, 12.34, 11.74 [10:17:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.60, 11.91, 11.65 [10:19:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.36, 12.88, 12.03 [10:22:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.96, 11.96, 11.89 [10:26:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.89, 12.08, 11.95 [11:03:00] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:01:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:15:03] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 22.79, 9.87, 4.54 [12:22:49] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.74, 3.08, 3.31 [12:26:41] PROBLEM - arru.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - arru.xyz All nameservers failed to answer the query. [12:33:12] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:46:09] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.55, 3.69, 2.89 [12:48:05] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.22, 2.77, 2.64 [12:55:53] RECOVERY - arru.xyz - reverse DNS on sslhost is OK: SSL OK - arru.xyz reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [12:58:52] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:00:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.113 second response time [13:02:35] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:03:32] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:41] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:43] PROBLEM - swiftobject122 SSH on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:23] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:38] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:28] PROBLEM - swiftobject122 ferm_active on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:29] PROBLEM - swiftobject122 conntrack_table_size on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:05:57] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:06:20] PROBLEM - swiftobject122 PowerDNS Recursor on swiftobject122 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:11:36] PROBLEM - swiftobject122 NTP time on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:38] RECOVERY - swiftobject122 conntrack_table_size on swiftobject122 is OK: OK: nf_conntrack is 0 % full [13:11:38] RECOVERY - swiftobject122 ferm_active on swiftobject122 is OK: OK ferm input default policy is set [13:11:38] RECOVERY - swiftobject122 PowerDNS Recursor on swiftobject122 is OK: DNS OK: 1.640 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [13:11:40] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 48 minutes ago with 0 failures [13:11:48] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.226 second response time [13:11:55] RECOVERY - swiftobject122 SSH on swiftobject122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [13:12:36] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.037 second response time [13:13:29] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.019 second response time [13:13:39] RECOVERY - swiftobject122 NTP time on swiftobject122 is OK: NTP OK: Offset 0.0007537603378 secs [13:14:28] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.024 second response time [13:22:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.05, 4.58, 2.73 [13:24:32] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.04, 5.77, 3.39 [13:26:32] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.88, 5.45, 3.60 [13:33:59] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:39:16] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1656766236 secs [13:46:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.90, 3.87, 2.84 [13:48:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 3.14, 2.70 [13:50:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:03:23] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:09:16] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05105543137 secs [14:14:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.64, 6.11, 7.85 [14:15:30] PROBLEM - jobchron121 APT on jobchron121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:17:33] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 53 packages available for upgrade (8 critical updates). [14:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.68, 7.26, 7.76 [14:22:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 6.85, 7.55 [14:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.27, 7.44, 7.38 [14:34:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.11, 7.28 [14:37:32] PROBLEM - phab121 APT on phab121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:38:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1724927127 secs [14:39:32] PROBLEM - phab121 APT on phab121 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (8 critical updates). [14:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.43, 7.62, 7.34 [14:54:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.96, 6.98, 7.14 [14:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.21, 7.13, 7.16 [14:58:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.06, 7.13 [15:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.99, 8.50, 7.67 [15:02:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.86, 6.23, 5.57 [15:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.36, 7.81, 7.53 [15:04:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.86, 5.67, 5.44 [15:05:37] PROBLEM - talumn.neptune.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'talumn.neptune.wiki' expires in 15 day(s) (Mon 19 Jun 2023 14:58:10 GMT +0000). [15:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.60, 8.08, 7.71 [15:07:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:08:31] PROBLEM - www.obeymewiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'obeymewiki.com' expires in 15 day(s) (Mon 19 Jun 2023 14:53:22 GMT +0000). [15:08:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 7.66, 7.61 [15:10:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.05919137597 secs [15:11:07] PROBLEM - obeymewiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'obeymewiki.com' expires in 15 day(s) (Mon 19 Jun 2023 14:53:22 GMT +0000). [15:12:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d5ed0028ca54...cf029e1be0bd [15:12:45] [02miraheze/ssl] 07MirahezeSSLBot 03cf029e1 - Bot: Update SSL cert for talumn.neptune.wiki [15:13:51] PROBLEM - wiki.aetherexplorers.science - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aetherexplorers.science' expires in 15 day(s) (Mon 19 Jun 2023 14:57:05 GMT +0000). [15:14:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cf029e1be0bd...a63efb9c7d81 [15:14:44] [02miraheze/ssl] 07MirahezeSSLBot 03a63efb9 - Bot: Update SSL cert for obeymewiki.com [15:18:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a63efb9c7d81...3f8f23ca4fef [15:18:38] [02miraheze/ssl] 07MirahezeSSLBot 033f8f23c - Bot: Update SSL cert for wiki.aetherexplorers.science [15:20:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.53, 7.56, 7.43 [15:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.39, 7.38 [15:27:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:34:43] RECOVERY - talumn.neptune.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'talumn.neptune.wiki' will expire on Fri 01 Sep 2023 14:12:36 GMT +0000. [15:36:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.21, 7.82, 7.40 [15:37:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [15:37:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.47, 3.38, 2.78 [15:37:32] RECOVERY - www.obeymewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'obeymewiki.com' will expire on Fri 01 Sep 2023 14:14:34 GMT +0000. [15:38:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.772 second response time [15:38:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.34, 6.86, 7.10 [15:39:19] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.74, 3.77, 2.99 [15:39:43] RECOVERY - obeymewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'obeymewiki.com' will expire on Fri 01 Sep 2023 14:14:34 GMT +0000. [15:43:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.57, 3.06, 2.93 [15:49:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.95, 7.82, 7.34 [15:52:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 7.02, 7.08 [15:55:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.927 second response time [15:56:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.53, 5.83, 6.59 [16:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 7.46, 7.10 [16:02:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.47, 7.15 [16:06:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.38, 8.11, 7.45 [16:09:16] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2654984295 secs [16:13:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:34] RECOVERY - wiki.aetherexplorers.science - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aetherexplorers.science' will expire on Fri 01 Sep 2023 14:18:27 GMT +0000. [16:17:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.707 second response time [16:18:44] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.39, 3.44, 2.85 [16:20:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.02, 3.26, 2.85 [16:22:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.71, 7.97, 7.95 [16:23:16] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.01643544436 secs [16:23:22] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:24] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.066 second response time [16:29:41] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.80, 7.82 [16:31:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.244 second response time [16:32:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.59, 6.97, 7.51 [16:36:02] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:19] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.93, 3.41, 2.86 [16:42:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.59, 6.26, 6.78 [16:43:19] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.85, 3.20, 2.85 [16:44:07] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.603 second response time [16:46:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.69, 7.29 [16:48:42]