[00:01:28] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.83, 2.78, 2.03 [00:03:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.49, 6.77, 6.16 [00:07:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 13.50, 7.16, 3.71 [00:07:19] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.44, 3.50, 1.83 [00:07:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.68, 7.11, 7.87 [00:07:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 4.44, 5.92, 5.97 [00:08:03] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.63, 7.97 [00:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.49, 7.88 [00:12:11] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 6.383 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [00:13:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.19, 7.48, 7.81 [00:15:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.34, 7.72 [00:15:24] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: DISK CRITICAL - free space: / 4255 MB (5% inode=89%); [00:17:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.58, 7.93, 6.13 [00:18:41] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:18:58] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.88, 3.56, 2.30 [00:19:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.13, 6.28, 5.74 [00:19:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.42, 3.09, 3.88 [00:20:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.94, 3.50, 2.42 [00:21:19] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.56, 3.83, 3.58 [00:23:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.62, 4.44, 4.20 [00:23:20] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.88, 3.15, 3.36 [00:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 15.79, 9.21, 8.12 [00:26:44] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.52, 4.26, 3.12 [00:27:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.05, 3.65, 3.97 [00:29:02] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:29:04] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:30:36] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.39, 3.74, 3.18 [00:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.14, 7.97, 7.90 [00:32:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.35, 2.94, 2.96 [00:32:34] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [00:33:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.18, 7.27, 6.09 [00:33:18] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.87, 3.35, 3.23 [00:33:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.50, 3.59, 3.78 [00:35:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.73, 6.78, 6.06 [00:37:18] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.35, 3.39, 3.27 [00:37:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 7.69, 7.74 [00:43:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.64, 3.81 [00:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.66, 7.92, 7.95 [00:48:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.47, 3.06, 2.76 [00:48:06] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 1213 bytes in 0.035 second response time [00:49:07] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.092 second response time [00:49:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.70, 6.99, 6.40 [00:49:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.35, 3.43, 3.63 [00:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.39, 9.46, 8.52 [00:49:30] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:34] PROBLEM - db112 MariaDB on db112 is CRITICAL: Can't connect to MySQL server on 'db112.miraheze.org' (115) [00:49:38] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [00:49:46] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:49:55] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [00:50:26] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.80, 3.25, 2.53 [00:51:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.46, 5.94, 6.07 [00:51:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 2.91, 3.40 [00:51:48] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [00:51:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 3.20, 2.94 [00:52:27] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.26, 3.63, 2.74 [00:54:08] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 4.00, 3.54, 3.31 [00:54:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.90, 3.27, 2.71 [00:55:18] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14335 MB (10% inode=99%); [00:56:04] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.01, 3.81, 3.44 [00:56:23] [02ManageWiki] 07The-Voidwalker closed pull request 03#411: testing alternative CI options - 13https://github.com/miraheze/ManageWiki/pull/411 [00:56:26] [02miraheze/ManageWiki] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [00:56:27] [02ManageWiki] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/ManageWiki [00:56:59] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.216 second response time [00:57:07] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20261 bytes in 0.185 second response time [00:57:34] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 90 Threads: 8 Questions: 45578 Slow queries: 662 Opens: 182 Open tables: 176 Queries per second avg: 506.422 [00:57:37] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [00:57:38] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 1.4%Current connections: 7 [00:57:55] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.122 second response time [00:58:10] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.97, 8.54, 7.12 [00:58:18] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 6.27, 4.17, 2.16 [00:59:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.93, 7.33, 7.85 [00:59:21] miraheze/ManageWiki - OAuthority the build has errored. [00:59:52] miraheze/mw-config - OAuthority the build passed. [01:00:14] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 2.48, 3.53, 2.17 [01:02:14] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 1.04, 2.61, 1.99 [01:09:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.29, 7.64, 7.65 [01:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.32, 7.43 [01:13:22] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.72, 7.95, 7.65 [01:14:49] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:14:49] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.71, 3.22, 2.02 [01:16:48] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.30, 2.95, 2.08 [01:16:50] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 2.670 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [01:17:24] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 9482 MB (13% inode=89%); [01:23:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.40, 7.43, 7.86 [01:23:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.85, 7.82 [01:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 8.46, 8.04 [01:25:21] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.23, 3.61, 3.94 [01:26:08] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:27:03] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:27:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.50, 7.34, 7.67 [01:29:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.97, 7.00, 7.47 [01:33:18] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.18, 2.35, 3.27 [01:33:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.63, 5.09, 6.63 [01:37:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.67, 7.06, 7.82 [01:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.24, 7.74, 7.87 [01:43:24] PROBLEM - mwtask141 Disk Space on mwtask141 is WARNING: DISK WARNING - free space: / 7759 MB (10% inode=89%); [01:44:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.82, 4.13, 3.37 [01:47:41] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.66, 3.85, 2.96 [01:48:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.62, 3.35 [01:49:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.80, 3.56, 2.97 [01:50:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.14, 4.10, 3.57 [01:51:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.63, 3.36, 2.97 [01:54:23] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.93, 3.99, 3.63 [01:56:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:56:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.65, 4.06, 3.68 [01:57:02] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:57:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.01, 3.61, 3.22 [01:58:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.59, 3.52, 3.53 [01:59:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.94, 3.02 [02:02:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.61, 3.77, 3.61 [02:06:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.15, 6.70, 6.27 [02:07:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 7.57, 7.97 [02:08:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.43, 6.17, 6.13 [02:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 8.22, 8.17 [02:09:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.81, 4.58, 3.66 [02:11:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.77, 3.88, 3.51 [02:11:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.55, 3.48, 3.71 [02:13:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.07, 4.43, 3.75 [02:13:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.00, 4.79, 4.15 [02:17:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.93, 7.07, 7.84 [02:17:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.22, 3.87, 3.67 [02:17:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.80, 3.96, 3.93 [02:19:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.74, 4.54, 3.96 [02:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 7.84, 7.95 [02:21:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.21, 2.56, 3.37 [02:21:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.06, 3.89, 3.78 [02:27:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:31:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.48, 4.37, 3.85 [02:33:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 7.66, 7.96 [02:35:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.90, 4.43, 3.74 [02:38:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1043393612 secs [02:39:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.69, 3.37, 3.51 [02:39:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.02, 3.21, 3.53 [02:40:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09329849482 secs [02:41:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.75, 2.49, 3.17 [02:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.12, 7.72, 7.71 [02:41:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 2.94, 3.39 [02:43:24] PROBLEM - mwtask141 Disk Space on mwtask141 is CRITICAL: DISK CRITICAL - free space: / 4126 MB (5% inode=89%); [02:45:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.65, 7.74 [02:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.76, 8.85, 8.16 [02:49:24] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 18032 MB (25% inode=91%); [02:55:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.98, 7.06, 7.70 [02:57:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.24, 8.42, 8.10 [02:59:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.91, 7.95 [03:03:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 11.94, 6.89, 4.40 [03:05:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.54, 8.24, 8.01 [03:09:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.79, 3.76, 3.28 [03:09:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.85, 3.65, 3.76 [03:11:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 3.05, 3.07 [03:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.50, 7.28, 7.75 [03:13:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:13:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.22, 2.43, 3.25 [03:17:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.13, 7.41 [03:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 8.00, 7.71, 7.59 [03:25:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.05, 4.04, 3.32 [03:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.90, 7.68 [03:28:26] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.92, 3.24, 1.94 [03:29:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 7.68, 7.68 [03:30:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.08, 2.77, 1.93 [03:31:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.35, 3.83, 3.59 [03:41:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.37, 2.63, 3.17 [03:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 7.45, 7.32 [03:43:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:43:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 6.83, 7.10 [03:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.27, 6.81, 6.95 [03:49:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.40, 6.37, 6.77 [03:53:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 6.93, 6.94 [03:53:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 3.46, 2.86 [03:55:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.27, 2.95, 2.73 [03:59:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.74, 7.81, 7.29 [04:03:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.15, 6.88, 7.08 [04:09:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.49, 6.23, 6.70 [04:13:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.93, 6.90 [04:17:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.32, 8.09, 7.35 [04:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.49, 6.98, 7.28 [04:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 7.13, 7.16 [04:33:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.15, 7.16 [04:43:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.09, 6.28, 6.73 [04:50:24] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:51:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 7.30, 6.92 [04:52:25] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [04:55:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.77, 7.70, 7.20 [04:59:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.53, 6.30, 6.77 [05:21:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.29, 6.42, 6.16 [05:23:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.20, 6.10 [05:27:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.17, 6.56 [05:33:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.97, 6.19, 6.36 [05:39:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 6.59, 6.41 [05:41:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.46, 6.52, 6.40 [05:54:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.12, 6.79 [05:56:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.79, 6.71 [06:05:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.83, 4.23, 2.35 [06:07:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.30, 4.12, 2.51 [06:15:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.23, 6.44, 3.83 [06:17:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 7.37, 6.70 [06:19:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.95, 6.63, 4.72 [06:19:19] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [06:21:23] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [06:23:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.16, 7.39, 7.04 [06:25:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.74, 6.13, 6.62 [06:29:29] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.58, 4.29, 2.70 [06:30:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.58, 10.36, 11.95 [06:30:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 10.93, 11.89 [06:30:25] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.76, 7.18, 6.86 [06:32:30] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [06:33:21] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.21, 3.14, 2.63 [06:34:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.25, 12.14, 12.15 [06:34:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.44, 11.20, 11.76 [06:35:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.52, 7.12 [06:38:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 11.32, 11.90 [06:38:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 10.95, 11.58 [06:40:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.80, 12.48, 12.04 [06:41:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.38, 6.54, 6.80 [06:44:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.91, 11.85, 11.95 [06:45:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.31, 7.03 [06:46:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.26, 11.02, 11.45 [06:46:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.64, 12.76, 12.26 [06:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.15, 7.00 [06:49:19] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [06:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.68, 7.21 [06:51:22] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [06:55:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.34, 7.11, 7.30 [07:07:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.00, 7.75, 7.42 [07:11:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 7.47, 7.40 [07:15:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.85, 8.10, 7.64 [07:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.73, 7.97 [07:35:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 7.93, 7.98 [07:43:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.82, 7.96 [07:47:17] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.70, 8.93, 8.24 [07:52:15] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [08:01:11] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:03:11] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [08:03:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.78, 7.43, 7.91 [08:04:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1448218226 secs [08:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.31, 7.68, 7.83 [08:13:25] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864196 seconds old and 1879 bytes [08:19:43] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [08:21:46] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [08:24:05] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08514043689 secs [08:29:03] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [08:33:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.49, 7.92 [08:35:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.57, 7.88 [08:37:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.13, 7.06, 7.66 [08:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.69, 7.45, 7.64 [08:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.67, 7.85, 7.87 [08:57:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.16, 7.42 [08:57:37] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:58:33] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.85, 3.49, 2.60 [09:00:29] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.39, 3.15, 2.59 [09:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.59, 7.25, 7.64 [09:15:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.77, 7.94, 7.72 [09:17:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.73, 7.66 [09:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.60, 8.67, 8.01 [09:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.45, 7.93, 7.96 [09:27:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.93, 7.94 [09:29:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.94, 8.00, 7.98 [09:29:30] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.84, 7.90 [09:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.49, 7.15, 7.85 [09:49:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 7.74, 7.97 [09:51:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 7.84, 7.99 [09:53:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.10, 8.43, 8.22 [09:57:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 7.67, 8.00 [09:59:04] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:13:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.82, 8.32, 7.67 [10:15:58] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.38, 2.96, 1.80 [10:17:18] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:17:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 7.32, 7.43 [10:17:58] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.13, 2.68, 1.84 [10:19:14] RECOVERY - cp22 APT on cp22 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [10:25:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.74, 8.25, 7.55 [10:27:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:27:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.94, 7.53 [10:27:37] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.91, 7.82, 7.53 [10:33:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 7.66, 7.51 [10:43:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.23, 8.08, 7.65 [10:49:13] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:51:16] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [10:55:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.83, 7.89 [10:57:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.65, 8.24, 8.07 [11:02:58] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [11:06:58] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [11:09:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 7.69, 7.94 [11:13:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [11:15:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.57, 7.75 [11:21:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.80, 7.77 [11:26:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:29:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 7.55, 7.58 [11:31:08] PROBLEM - wiki.thedev.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thedev.gq could not be found [11:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.64, 7.61 [11:33:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.70, 8.33, 7.87 [11:43:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:51:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:51:59] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 12.53, 17.15, 8.92 [11:57:44] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 1.99, 6.49, 6.62 [11:58:34] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.07, 5.21, 3.30 [12:02:27] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.85, 3.72, 3.14 [12:04:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.28, 2.87, 2.89 [12:11:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.468 second response time [12:29:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.73, 4.00, 2.86 [12:31:10] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 3.08, 2.66 [12:39:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 7.33, 7.99 [12:41:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.65, 8.02 [13:00:24] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.88, 6.71, 6.43 [13:02:18] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.26, 6.61, 6.43 [13:06:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.29, 6.97, 6.61 [13:08:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.18, 6.42, 6.46 [13:09:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:25:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.14, 6.43, 6.20 [13:27:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.54, 5.99, 6.06 [13:31:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.59, 6.37, 3.36 [13:33:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.25, 6.45, 3.77 [13:49:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.591 second response time [13:53:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.87, 4.15, 2.78 [13:56:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.38, 3.33, 2.76 [14:00:54] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.998 second response time [14:01:54] RECOVERY - o.kaoyu666.cn - reverse DNS on sslhost is OK: SSL OK - o.kaoyu666.cn reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:09:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:11:31] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.722 second response time [14:29:04] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:36:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:41:25] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.45, 3.18, 2.21 [14:43:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 2.78, 2.18 [14:48:04] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.121689111 secs [14:50:55] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.258 second response time [14:57:37] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [14:59:33] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:09:19] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.19, 3.21, 1.75 [15:12:04] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09377288818 secs [15:12:11] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.467 second response time [15:15:18] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.67, 3.30, 2.39 [15:17:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.11, 6.58, 5.59 [15:19:17] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.31, 5.59, 5.33 [15:20:38] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:44] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.402 second response time [15:27:03] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:29:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.30, 7.09, 6.67 [15:29:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.21, 6.73, 6.59 [15:34:05] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.470 second response time [15:35:13] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.62, 3.56, 2.59 [15:36:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:37:18] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.305 second response time [15:39:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.41, 3.41, 2.77 [15:41:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 2.80, 2.62 [15:44:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.41, 2.95 [15:53:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.97, 3.12, 2.89 [15:57:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.484 second response time [16:06:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.69, 6.95, 6.29 [16:08:34] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.84, 6.60, 6.24 [16:37:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.53, 4.14, 3.40 [16:39:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 3.31, 3.18 [16:46:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:47:40] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:50:01] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.17, 4.33, 3.01 [16:50:13] RECOVERY - cp23 APT on cp23 is OK: APT OK: 34 packages available for upgrade (0 critical updates). [16:51:18] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.62, 6.20, 4.51 [16:53:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.02, 6.85, 4.96 [16:55:18] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.25, 5.09, 4.54 [16:57:39] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.62, 3.55, 3.31 [16:59:35] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.09, 3.30, 3.24 [17:00:26] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.55, 3.19, 2.36 [17:02:26] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.50, 2.48, 2.19 [17:13:18] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.41, 3.86, 3.36 [17:14:56] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:16:06] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 15.92, 10.28, 6.38 [17:17:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:17:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.60, 3.72, 2.84 [17:18:58] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.760 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [17:19:19] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.53, 3.73, 3.56 [17:19:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.70, 3.46, 2.84 [17:19:35] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [17:21:18] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.86, 3.07, 3.33 [17:21:37] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [17:21:48] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.43, 6.69, 6.12 [17:23:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.56, 4.02, 3.19 [17:25:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.50, 3.48, 3.09 [17:27:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.17, 3.87, 3.26 [17:35:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.88, 3.64, 3.60 [17:37:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.79, 4.51, 3.91 [17:38:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.41, 6.99, 6.43 [17:40:44] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.46, 6.72, 6.39 [17:41:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.95, 3.56, 3.69 [17:45:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.20, 2.91, 3.40 [17:50:11] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:19] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.317 second response time [17:58:09] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.31, 3.16, 2.62 [18:00:05] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.18, 2.40, 2.41 [18:00:49]