[00:00:27] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.654 second response time [00:01:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 11.07, 6.10, 3.72 [00:02:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.56, 2.86 [00:04:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.16, 3.77, 3.01 [00:04:58] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.25, 3.59, 3.24 [00:06:51] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:56] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.70, 2.60, 2.91 [00:08:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.36, 3.04 [00:08:55] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.562 second response time [00:09:44] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.66, 3.48, 2.07 [00:13:45] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.80, 3.51, 2.42 [00:14:29] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.57, 6.87, 4.84 [00:15:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.08, 2.98, 2.36 [00:15:45] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1171976626 secs [00:16:26] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.96, 6.46, 4.93 [00:16:41] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:16:44] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.65, 2.70, 2.63 [00:17:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:42] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.43, 2.12, 2.42 [00:19:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.401 second response time [00:19:42] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.53, 2.70, 3.99 [00:21:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:22:14] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [00:23:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 3.98, 3.51 [00:24:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.11, 3.78, 3.51 [00:25:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.43, 7.17, 5.66 [00:25:59] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:07] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.53, 7.22, 5.85 [00:27:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 2.32, 3.28 [00:27:45] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.09548082948 secs [00:28:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 3.25, 3.39 [00:29:04] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.79, 7.94, 6.26 [00:29:28] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:31:47] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.20, 4.30, 3.26 [00:32:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.686 second response time [00:35:59] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:44] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:58] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.901 second response time [00:40:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.58, 2.70, 2.19 [00:42:18] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 2.26, 2.09 [00:55:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 3.54, 2.89 [00:56:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:57:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.77, 3.24, 2.86 [00:57:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.49, 6.90, 7.98 [00:59:59] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.70, 3.90, 2.73 [01:00:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:59] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:01:32] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406410217285156 seconds [01:01:57] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.31, 3.92, 2.90 [01:02:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.987 second response time [01:03:01] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [01:03:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.91, 3.19, 2.75 [01:05:46] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.12, 3.38, 3.92 [01:06:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.681 second response time [01:07:45] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.17, 3.94, 4.05 [01:13:42] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.59, 3.58, 3.95 [01:15:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.175 second response time [01:17:45] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.77, 4.40, 4.15 [01:18:49] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.29, 11.41, 9.18 [01:21:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.62, 7.43 [01:22:50] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 7.09, 9.19, 8.78 [01:24:20] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:24:34] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:45] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.80, 3.58, 3.91 [01:26:38] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.862 second response time [01:27:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:29:26] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.48, 2.99, 2.52 [01:31:25] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.73, 3.80, 2.89 [01:31:27] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:31:45] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.04, 3.49, 3.71 [01:35:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [01:35:21] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.48, 3.17, 2.91 [01:35:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.95, 7.96 [01:35:46] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.65, 3.60, 3.72 [01:37:46] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.98, 4.13, 3.87 [01:40:24] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:57] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:41:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.90, 8.36, 8.04 [01:42:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.967 second response time [01:45:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 0.96, 2.87, 3.60 [01:47:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.87, 6.04, 7.93 [01:47:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.34, 2.47, 3.37 [01:53:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:53:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.37, 4.92, 6.76 [01:53:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.77, 7.23, 7.80 [01:53:56] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:57:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.59, 7.44, 7.69 [01:59:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.04, 7.52 [02:01:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.45, 3.55, 2.63 [02:06:26] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.30, 8.18, 7.08 [02:07:40] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [02:07:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.52, 2.78, 2.62 [02:11:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.44, 5.48, 6.50 [02:12:16] PROBLEM - mirawww.glitchcity.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mirawww.glitchcity.wiki All nameservers failed to answer the query. [02:15:34] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:16:12] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 1.58, 5.88, 7.10 [02:17:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.73, 7.24, 7.02 [02:18:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.69, 4.18, 6.34 [02:23:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [02:23:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.59, 8.60, 7.60 [02:33:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.50, 7.57 [02:40:51] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:41:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 6.91, 7.12 [02:42:13] PROBLEM - mirawww.glitchcity.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mirawww.glitchcity.wiki could not be found [02:43:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 6.70, 7.01 [02:44:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:14] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:45:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.92, 7.53, 7.27 [02:47:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.48, 7.28 [02:50:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.074 second response time [02:52:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:57:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.08, 6.74 [02:57:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.32, 3.93, 3.11 [02:59:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.71, 4.27, 3.33 [02:59:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.63, 4.01, 2.57 [03:00:45] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 44 seconds old and 275 bytes [03:01:05] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:01:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.99, 3.82, 3.27 [03:02:34] PROBLEM - db112 Backups SQL on db112 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864152 seconds old and 365 bytes [03:03:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.70, 5.69, 3.62 [03:03:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 6.71, 6.82 [03:03:45] PROBLEM - db112 Backups SQL on db112 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864222 seconds old and 365 bytes [03:05:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 3.16, 3.15 [03:08:54] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.23, 3.92, 3.52 [03:10:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.28, 3.38, 3.37 [03:11:55] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [03:13:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.29, 8.32, 7.51 [03:13:40] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.96, 3.48, 3.73 [03:15:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.94, 7.48 [03:15:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.61, 2.51, 3.35 [03:16:52] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:20:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:25:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 7.65, 7.32 [03:29:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.61, 7.40 [03:39:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.79, 5.75, 6.62 [03:43:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.14, 7.02 [03:45:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:46:33] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:51:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.50, 7.76, 7.33 [03:52:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.18, 3.42, 2.87 [03:53:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.97, 3.67, 3.15 [03:53:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.52, 7.31 [03:54:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 2.86, 2.72 [03:55:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.02, 3.10, 3.00 [03:59:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.83, 7.94, 7.45 [04:03:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 7.69, 7.47 [04:17:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:19:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.24, 8.18, 7.82 [04:20:11] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:22:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:24:35] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7964 MB (5% inode=99%); [04:25:22] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [04:25:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.68, 7.79 [04:29:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.96, 7.84 [04:33:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 7.65, 7.71 [04:35:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.81, 7.90, 7.78 [04:39:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.77, 7.75 [04:41:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.80, 5.64, 3.03 [04:41:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.95, 8.21, 7.91 [04:41:44] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.64, 3.70, 1.85 [04:43:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.07, 5.65, 3.38 [04:43:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.95, 3.48, 2.01 [04:45:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.86, 3.00, 2.02 [04:47:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.00, 7.64, 7.82 [04:48:29] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:48:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.97, 3.62, 3.04 [04:52:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 3.19, 3.01 [04:57:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.17, 7.33 [04:59:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 7.31, 7.36 [05:00:06] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.82, 3.36, 2.40 [05:00:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:00:48] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.86, 6.35, 6.08 [05:02:04] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.28, 2.63, 2.25 [05:02:13] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [05:04:47] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.57, 6.51, 6.20 [05:17:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 6.99, 7.11 [05:23:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 7.96, 7.55 [05:27:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.58, 8.19, 7.73 [05:31:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 7.89, 7.73 [05:34:27] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.15, 6.60, 6.28 [05:36:26] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.03, 6.42, 6.26 [05:36:32] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.59, 2.76 [05:38:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.81, 4.30, 3.13 [05:40:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 3.33, 2.91 [05:41:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 7.43, 7.45 [05:43:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.27, 7.38 [05:47:33] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:49:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 7.93, 7.61 [05:53:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.00, 7.50, 7.58 [05:57:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.78, 7.66 [05:59:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 7.52, 7.56 [06:01:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.46, 7.96, 7.72 [06:03:45] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1748560369 secs [06:05:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.95, 7.78 [06:07:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 8.06, 7.85 [06:11:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 7.74, 7.81 [06:13:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:17:30] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:19:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.75, 8.24, 7.89 [06:25:45] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.06224286556 secs [06:26:12] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.034 second response time [06:26:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.46, 4.38, 2.96 [06:28:34] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.76, 5.40, 3.16 [06:28:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.92, 3.71, 2.87 [06:29:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.88, 7.24, 7.80 [06:30:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:30:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.26, 3.24, 2.79 [06:31:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.43, 8.00, 7.99 [06:32:30] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.58, 3.79, 3.07 [06:34:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.46, 3.16, 2.93 [06:44:09] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:45:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.67, 7.29, 8.00 [06:49:27] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:49:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.31, 7.82 [06:51:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 7.07, 7.69 [06:52:00] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [06:57:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.25, 5.35, 2.94 [06:57:45] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1193301082 secs [06:59:17] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.606 second response time [06:59:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.53, 5.58, 3.34 [06:59:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 8.03, 7.76 [07:00:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:11:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 7.67, 7.96 [07:13:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.42, 8.59, 8.24 [07:13:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:45] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09851062298 secs [07:15:38] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.992 second response time [07:19:17] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [07:19:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:21:18] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [07:23:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:30:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:30:35] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.72, 3.45, 2.51 [07:32:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.44, 3.12, 2.51 [07:33:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.61, 6.70, 6.25 [07:39:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.81, 6.50, 6.33 [07:43:30] PROBLEM - wonderfuleveryday.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wonderfuleveryday.org' expires in 15 day(s) (Tue 04 Jul 2023 07:35:37 GMT +0000). [07:49:10] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [07:49:21] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:49:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c9849b952655...d28cf35f0524 [07:49:55] [02miraheze/ssl] 07MirahezeSSLBot 03d28cf35 - Bot: Update SSL cert for wonderfuleveryday.org [07:51:10] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [07:52:45] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:58:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:00:42] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.420 second response time [08:08:32] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 1 backends are down. mw122 [08:10:32] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [08:12:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.367 second response time [08:15:40] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:20:41] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [08:31:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.62, 7.98 [08:33:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.85, 8.02 [08:36:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.93, 6.71, 6.48 [08:37:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.73, 7.98 [08:38:31] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.79, 6.29, 6.35 [08:39:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.90, 9.60, 8.65 [08:42:31] RECOVERY - wonderfuleveryday.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wonderfuleveryday.org' will expire on Sat 16 Sep 2023 06:49:45 GMT +0000. [08:43:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:50:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:53:59] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.683 second response time [08:54:54] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.796 second response time [08:57:40] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.50, 3.72, 2.74 [08:59:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.03, 3.36, 2.75 [09:04:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:15:06] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.16, 6.76, 6.40 [09:17:04] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.34, 6.40, 6.32 [09:18:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.55, 6.60, 5.28 [09:20:21] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.18, 5.75, 5.14 [09:29:00] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:35:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:35:13] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.633 second response time [09:43:46] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:49] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.971 second response time [09:48:17] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:49:12] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [09:51:13] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [09:51:36] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:56:45] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [09:57:21] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.57, 3.65, 2.89 [09:59:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.27, 3.21, 2.83 [10:03:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.03, 7.92 [10:11:34] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:12:05] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.90, 4.81, 3.54 [10:13:42] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [10:14:03] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.15, 3.99, 3.38 [10:15:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.76, 7.89, 7.70 [10:16:01] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.33, 3.37, 3.22 [10:17:32] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:45] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1809431612 secs [10:17:57] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:19:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.199 second response time [10:22:06] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:25:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.33, 7.29, 7.78 [10:29:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.53, 7.73 [10:31:44] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.17, 3.41, 2.86 [10:32:56] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:35:03] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [10:35:25] PROBLEM - www.lab612.at - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lab612.at All nameservers failed to answer the query. [10:35:40] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.95, 3.68, 3.19 [10:37:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 7.57, 7.73 [10:37:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.13, 2.82, 2.94 [10:39:45] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08887657523 secs [10:39:45] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4066197872161865 seconds [10:45:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.12, 7.63, 7.59 [10:47:38] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:49:14] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:51:14] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [10:55:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:05:05] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: SSL OK - www.lab612.at reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:05:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.77, 6.65, 7.88 [11:07:45] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1294310987 secs [11:09:40] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:13:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.54, 7.76, 7.87 [11:13:45] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.05901286006 secs [11:19:15] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [11:19:32] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:21:32] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:21:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.96, 8.00 [11:22:00] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:23:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.88, 8.08, 8.05 [11:35:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 7.68, 7.96 [11:37:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.72, 7.94 [11:39:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.69, 7.35, 7.77 [11:47:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.43, 7.25, 7.44 [11:48:53] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:51:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.86, 7.87, 7.68 [11:56:30] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.41, 4.34, 2.88 [11:58:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 3.32, 2.69 [12:01:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.14, 7.78, 7.53 [12:03:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.67, 7.52 [12:05:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.45, 6.73, 6.20 [12:07:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.82, 6.28, 6.10 [12:15:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.16, 8.15, 7.81 [12:17:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.52, 7.61 [12:19:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.73, 8.14, 7.81 [12:21:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:25:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 8.00, 7.84 [12:27:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.27, 8.19, 7.94 [12:28:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.22, 3.41, 2.34 [12:30:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.84, 3.10, 2.36 [12:34:23] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:49] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:40:36] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.067 second response time [12:41:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.62, 7.26, 7.84 [12:42:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.54, 8.19, 8.06 [12:46:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:50:10] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:50:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:46] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.900 second response time [12:57:01] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.692 second response time [12:58:25] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.31, 4.11, 2.65 [12:59:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.41, 7.23, 7.90 [13:00:23] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.99, 3.66, 2.67 [13:01:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.81, 9.35, 8.56 [13:02:22] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.71, 2.93, 2.52 [13:03:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.412 second response time [13:03:45] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1715895832 secs [13:05:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:09:51] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.47, 6.58, 3.51 [13:13:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.402 second response time [13:15:07] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw121 [13:15:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.60, 5.25, 3.82 [13:16:31] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:07] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [13:18:04] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw122 [13:18:27] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.788 second response time [13:18:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:19:49] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:20:03] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [13:20:13] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:21] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.682 second response time [13:25:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.66, 6.38, 5.73 [13:25:45] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09209161997 secs [13:26:04] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 1 backends are down. mw121 [13:26:36] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 12.79, 9.19, 7.26 [13:27:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.42, 4.65, 3.51 [13:28:03] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [13:29:01] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.88, 6.56, 6.00 [13:30:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.79, 7.60, 7.02 [13:30:42] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:59] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [13:31:11] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.636 second response time [13:31:51] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.18, 3.55, 3.40 [13:33:49] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.97, 2.71, 3.11 [13:35:20] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:37:38] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:29] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.45, 6.53, 6.75 [13:41:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.265 second response time [13:42:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.60, 6.99, 6.91 [13:43:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:43:45]