[00:02:27] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.20, 5.52, 3.80 [00:02:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.31, 4.31, 3.43 [00:02:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.49, 6.68, 6.08 [00:03:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:06:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.94, 6.46, 6.14 [00:07:01] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.72, 3.29, 1.88 [00:07:50] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:08:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.42, 6.69, 4.32 [00:08:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.12, 3.76, 3.60 [00:10:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.17, 7.07, 4.77 [00:12:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.37, 5.77, 4.58 [00:12:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.00, 3.33 [00:13:02] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.43, 3.82, 2.65 [00:14:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.66, 3.67, 3.85 [00:17:02] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.38, 3.96, 2.95 [00:18:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.92, 2.42, 3.31 [00:19:02] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.30, 3.34, 2.85 [00:23:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.62, 3.76, 3.49 [00:27:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.08, 3.40, 3.40 [00:30:53] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1228591204 secs [00:33:01] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.89, 4.27, 3.39 [00:33:11] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.44, 8.09, 6.04 [00:33:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:34:53] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08661448956 secs [00:34:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.39, 4.31, 3.64 [00:35:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.44, 7.19, 6.59 [00:36:42] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.09, 3.61, 3.45 [00:37:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.84, 6.93, 6.56 [00:38:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 3.12, 3.29 [00:40:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.547 second response time [00:40:51] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.97, 8.00, 7.05 [00:43:02] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.74, 3.82, 3.66 [00:43:30] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.98, 6.60, 6.57 [00:44:25] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.66, 7.85, 7.13 [00:46:27] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.394 second response time [00:46:36] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.70, 6.85, 6.86 [00:48:31] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.05, 6.05, 6.55 [00:49:03] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.28, 3.07, 3.36 [00:49:26] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:51:29] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [00:52:23] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.46, 7.53, 7.02 [00:53:09] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.47, 3.46, 3.46 [00:54:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.26, 5.84, 6.46 [00:55:07] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.26, 3.23, 3.37 [00:58:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.69, 5.02, 3.69 [00:59:03] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.19, 3.72, 3.53 [01:02:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.24, 7.79, 7.17 [01:02:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.03, 3.58, 3.40 [01:03:02] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.69, 3.92, 3.68 [01:03:09] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.83, 7.26, 7.04 [01:04:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.18, 4.27, 3.66 [01:06:53] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1093528569 secs [01:07:02] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.06, 3.90, 3.71 [01:08:53] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.08980834484 secs [01:09:01] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.70, 3.83, 3.71 [01:10:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.71, 7.43, 7.15 [01:10:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.46, 3.87, 3.73 [01:11:01] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.17, 4.30, 3.90 [01:12:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.21, 7.27, 7.12 [01:13:27] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.381 second response time [01:14:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 2.73, 3.27 [01:15:02] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.60, 3.90, 3.88 [01:17:03] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.18, 4.33, 4.03 [01:18:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.66, 8.01, 7.40 [01:21:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.01, 3.44, 3.46 [01:21:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.70, 3.21, 3.37 [01:23:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.164 second response time [01:31:37] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [01:33:36] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.025 second response time [01:35:56] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.81, 6.88, 6.47 [01:36:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.61, 6.17, 7.77 [01:37:52] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.41, 6.50, 6.37 [01:39:02] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.78, 3.48, 4.00 [01:41:04] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.78, 4.12, 4.15 [01:42:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.41, 5.03, 6.71 [01:43:04] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.40, 3.70, 3.99 [01:45:33] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [01:47:01] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.57, 2.20, 3.34 [01:48:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.66, 2.98 [01:49:32] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.021 second response time [01:50:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 2.75, 2.72 [01:50:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.98, 6.82, 6.57 [01:52:27] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.68, 6.23, 6.37 [02:04:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.66, 3.81, 3.15 [02:05:06] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.24, 7.02, 6.60 [02:06:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.48, 3.40, 3.08 [02:07:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.34, 6.83, 6.59 [02:07:50] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:07:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:10:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.44, 3.54, 3.26 [02:11:04] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:11:53] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.549 second response time [02:12:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.80, 3.38 [02:14:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 3.06, 3.15 [02:14:51] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.30, 6.67, 6.61 [02:16:21] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.57, 2.54, 1.58 [02:18:06] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:18:21] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.94, 2.55, 1.70 [02:20:05] RECOVERY - cp22 APT on cp22 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [02:36:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.75, 6.97, 6.66 [02:37:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.72, 4.35, 3.30 [02:39:10] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.16, 6.12, 4.34 [02:40:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.50, 6.80, 6.65 [02:41:05] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.70, 6.63, 4.79 [02:43:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.26, 3.83, 3.68 [02:45:12] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.567 second response time [02:46:14] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:46:51] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.44, 7.18, 5.41 [02:48:46] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.13, 7.65, 5.82 [02:49:57] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.99, 7.24, 6.81 [02:50:12] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.23, 3.07, 2.20 [02:50:41] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.07, 5.54, 5.26 [02:52:00] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [02:53:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.07, 6.63, 6.65 [02:54:09] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 2.79, 2.32 [02:54:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.22, 2.87, 3.33 [02:56:59] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:57:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.17, 6.90, 6.82 [02:58:06] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.14, 3.43, 2.70 [02:59:41] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.20, 6.66, 6.75 [03:00:04] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.60, 2.90, 2.59 [03:03:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.30, 4.29, 3.33 [03:06:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:54] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.75, 3.67, 3.29 [03:08:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.752 second response time [03:08:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.28, 3.16, 3.15 [03:11:38] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:21:15] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14600 MB (10% inode=99%); [03:22:26] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [03:26:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.86, 6.45, 6.42 [03:28:51] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.64, 6.45, 6.42 [03:36:38] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [03:40:31] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.028 second response time [03:42:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.78, 5.75, 3.19 [03:42:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:43:32] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:44:28] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.263 second response time [03:45:01] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.63, 3.03, 1.61 [03:45:48] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:46:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.26, 7.70, 4.66 [03:47:02] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.15, 2.79, 1.71 [03:48:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.24, 5.84, 4.34 [03:50:43] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [03:53:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.92, 6.84, 6.56 [03:55:49] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:57:18] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.38, 6.57, 6.51 [03:59:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.77, 3.37, 2.47 [04:01:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.14, 2.98, 2.42 [04:02:45] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [04:03:01] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:05:05] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.97, 6.95, 6.66 [04:06:33] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.035 second response time [04:07:02] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.54, 6.37, 6.48 [04:21:43] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:24:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.62, 4.10, 3.11 [04:26:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.90, 3.15 [04:28:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 3.37, 3.04 [04:29:59] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [04:32:37] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:38:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.78, 5.77, 3.95 [04:40:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.25, 5.39, 4.02 [04:42:32] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:44:26] PROBLEM - mwtask141 Disk Space on mwtask141 is WARNING: DISK WARNING - free space: / 6682 MB (9% inode=89%); [04:50:42] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:52:26] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.45, 3.26, 2.35 [04:52:40] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [04:54:24] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.78, 2.33, 2.11 [04:56:26] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: DISK CRITICAL - free space: / 3640 MB (5% inode=89%); [05:02:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.06, 6.93, 7.88 [05:04:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.62, 8.01 [05:09:50] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:15:27] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 2.05, 1.24, 0.72 [05:17:26] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.64, 1.02, 0.71 [05:19:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.56, 4.71, 3.28 [05:25:45] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.31, 3.53, 3.21 [05:27:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.24, 2.81, 2.99 [05:38:41] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:42:50] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6664 bytes in 0.074 second response time [05:42:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.12, 7.13, 7.89 [05:44:50] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.345 second response time [05:44:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.99, 8.44, 8.28 [05:50:49] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.501 second response time [05:59:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:00:35] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.87, 3.84, 2.58 [06:01:27] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.205 second response time [06:02:05] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:04:03] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [06:04:32] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.26, 2.98, 2.57 [06:04:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.76, 7.25, 7.97 [06:05:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:06:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.26, 8.41, 8.31 [06:07:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 4.13, 2.93 [06:09:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.155 second response time [06:09:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 3.22, 2.74 [06:11:04] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:18:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.93, 7.21, 7.76 [06:20:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.53, 7.80 [06:22:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.59, 6.63, 7.43 [06:26:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.33, 6.95, 6.39 [06:26:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.00, 7.34 [06:28:14] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 4.00, 3.48, 2.64 [06:28:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.13, 6.60, 6.32 [06:30:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:30:13] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.29, 3.57, 2.76 [06:30:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 7.63, 7.57 [06:32:12] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.05, 2.65, 2.52 [06:32:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.00, 8.40, 7.85 [06:38:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.344 second response time [06:49:08] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [06:49:49] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:10] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [06:51:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.759 second response time [06:56:09] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.258 second response time [07:19:26] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [07:21:30] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [07:46:15] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.48, 2.99, 2.38 [07:46:21] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.74, 2.76, 1.93 [07:46:44] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:48:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.78, 2.85, 2.40 [07:48:21] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.91, 2.44, 1.91 [07:49:27] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [07:50:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.99, 4.88, 2.95 [07:53:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.93, 3.88, 3.02 [07:55:06] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.67, 3.77, 2.97 [07:55:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.67, 3.08, 2.82 [07:57:05] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.28, 3.27, 2.89 [08:04:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.832 second response time [08:06:46] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.78, 5.97, 3.70 [08:08:41] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.82, 6.69, 4.25 [08:08:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.26, 6.91, 7.85 [08:10:36] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.38, 5.09, 3.95 [08:10:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.71, 7.97, 8.11 [08:18:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:22:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.911 second response time [08:26:46] [02puppet] 07paladox closed pull request 03#3225: remove paladox shell and icinga access - 13https://github.com/miraheze/puppet/pull/3225 [08:26:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/560212c418fb...ec999b36bed3 [08:26:50] [02miraheze/puppet] 07MacFan4000 03ec999b3 - remove paladox shell and icinga access (#3225) [08:28:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.81, 2.88, 2.30 [08:30:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.03, 3.28, 2.52 [08:32:39] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.68, 2.68, 2.39 [08:39:49] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:58:18] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.44, 3.99, 2.71 [08:59:20] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:00:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.05, 3.52, 2.69 [09:01:18] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [09:02:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.00, 2.99, 2.60 [09:08:38] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:26:47] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:27:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.61, 4.22, 2.79 [09:28:46] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [09:31:43] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.72, 3.74, 2.95 [09:32:29] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [09:34:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 6.83, 7.88 [09:34:27] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.030 second response time [09:35:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.62, 3.15, 2.91 [09:38:23] PROBLEM - db142 Backups SQL on db142 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864189 seconds old and 1879 bytes [09:38:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.00, 6.69, 6.34 [09:40:37] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.25, 6.34, 6.24 [09:41:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.01, 8.75, 8.22 [09:42:46] PROBLEM - wiki.mcsoft.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.407032012939453 seconds [09:57:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.13, 3.49, 2.62 [09:59:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.58, 6.51, 3.73 [10:01:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.72, 5.95, 3.89 [10:01:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.44, 3.39, 2.82 [10:09:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.70, 6.89, 7.81 [10:09:54] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:11:04] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [10:12:11] RECOVERY - wiki.mcsoft.org - reverse DNS on sslhost is OK: SSL OK - wiki.mcsoft.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:15:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.37, 2.97, 1.89 [10:17:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.09, 7.52 [10:19:10] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:19:38] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.98, 3.62, 2.47 [10:21:09] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [10:21:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.51, 2.82, 2.32 [10:21:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 7.78, 7.73 [10:24:25] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:25:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.63, 7.65 [10:30:31] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.56, 4.69, 3.27 [10:32:08] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.40, 7.96, 7.96 [10:34:06] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [10:34:27] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.92, 3.75, 3.25 [10:36:25] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.79, 3.01, 3.04 [10:37:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:34] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:43:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.867 second response time [10:45:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.70, 8.17, 7.90 [10:45:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:45:59] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.10, 3.77, 2.41 [10:46:58] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:47:55] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.87, 2.87, 2.24 [10:49:21] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [10:54:22] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:56:04] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [10:56:11] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.840 second response time [10:59:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:22] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:00:59] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.16, 4.03, 3.13 [11:01:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.69, 8.00 [11:02:20] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [11:02:57] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.36, 3.09, 2.89 [11:05:10] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [11:05:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.58, 7.52, 7.80 [11:06:04] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.192 second response time [11:07:10] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.034 second response time [11:09:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.94, 7.94 [11:11:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.43, 8.91, 8.30 [11:12:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.088 second response time [11:19:28] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:21:09] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:21:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:28] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:22:35] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.05, 4.69, 3.34 [11:26:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:26:30] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [11:28:29] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.93, 3.72, 3.39 [11:30:27] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.95, 3.10, 3.19 [11:33:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.441 second response time [11:35:26] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 503 Service Unavailable [11:36:24] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:20] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.061 second response time [11:37:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.86, 6.65, 6.15 [11:39:20] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.05, 6.43, 6.13 [11:40:35] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.795 second response time [11:50:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.08, 3.76, 2.94 [11:54:18] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.835 second response time [11:59:52] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.31, 3.29, 3.22 [12:01:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.810 second response time [12:06:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:25] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:12:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.043 second response time [12:18:31] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.76, 3.94, 2.75 [12:20:26] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.26, 2.98, 2.54 [12:23:30] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.41, 4.45, 3.66 [12:29:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:05] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.644 second response time [12:32:00] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [12:35:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.18, 3.09, 4.00 [12:36:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.31, 7.05, 6.60 [12:37:32] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:41] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.16, 6.66, 6.51 [12:39:17] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.75, 1.93, 3.32 [12:41:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.156 second response time [12:42:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:00] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [12:47:58] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.051 second response time [12:50:08] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:37] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:53:41] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [12:55:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.535 second response time [12:58:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.90, 4.31, 3.36 [13:00:19] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4063332080841064 seconds [13:00:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.115 second response time [13:00:40] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 13032 MB (18% inode=89%); [13:02:50] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.09, 3.73, 3.39 [13:04:48] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.47, 3.22, 3.23 [13:04:54] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.21, 6.47, 6.24 [13:08:23] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.49, 6.43, 6.25 [13:08:57] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.818 second response time [13:12:41] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:25] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.61, 3.82, 3.30 [13:26:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:26:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.90, 6.83, 6.54 [13:28:09] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.88, 6.69, 6.53 [13:29:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.93, 3.37, 3.30 [13:35:35] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:42] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:41:40] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.14, 6.87, 3.79 [13:41:46] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.198 second response time [13:42:51] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [13:43:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.54, 5.23, 3.58 [13:44:25] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.177 second response time [13:50:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.10, 3.00, 2.59 [13:54:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.74, 2.97, 2.64 [14:00:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.77, 4.62, 3.50 [14:04:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.97, 3.96, 3.54 [14:05:45] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1196260452 secs [14:05:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.308 second response time [14:06:39] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.95, 3.32, 3.36 [14:07:45] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08729985356 secs [14:08:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:36] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.086 second response time [14:11:01] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [14:15:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.44, 4.44, 2.90 [14:15:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.25, 9.45, 5.37 [14:15:51] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.84, 3.94, 2.18 [14:17:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.16, 6.48, 4.77 [14:17:40]