[00:17:29] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6bb5b9a2d6b2 [00:17:30] [02miraheze/mw-config] 07MacFan4000 036bb5b9a - Update LocalSettings.php [00:17:31] [02mw-config] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/mw-config [00:17:51] [02mw-config] 07MacFan4000 opened pull request 03#5251: disallow wiki creations and requests - 13https://github.com/miraheze/mw-config/pull/5251 [00:18:52] miraheze/mw-config - MacFan4000 the build passed. [00:27:13] [02mw-config] 07MacFan4000 closed pull request 03#5251: disallow wiki creations and requests - 13https://github.com/miraheze/mw-config/pull/5251 [00:27:14] [02miraheze/mw-config] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [00:27:15] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/33f6d30ba7de...048d2f3f7761 [00:27:18] [02miraheze/mw-config] 07MacFan4000 03048d2f3 - disallow wiki creations and requests (#5251) [00:27:21] [02mw-config] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/mw-config [00:28:17] miraheze/mw-config - MacFan4000 the build passed. [00:30:35] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:30:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:26] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 171s [00:33:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:03] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=MacFan4000 (END - exit=0) [00:37:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:55] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/048d2f3f7761...c915892162c2 [00:43:56] [02miraheze/mw-config] 07MacFan4000 03c915892 - Update LocalSettings.php [00:43:57] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:44:07] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:44:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:57] miraheze/mw-config - MacFan4000 the build passed. [00:57:27] !log [@test131] starting deploy of {'config': True} to all [00:57:28] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [00:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:05] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c915892162c2...5e6dea2d74fc [02:09:06] [02miraheze/mw-config] 07MacFan4000 035e6dea2 - try again to disable wiki requests [02:09:57] miraheze/mw-config - MacFan4000 the build has errored. [02:10:35] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5e6dea2d74fc...bfb38224fb58 [02:10:36] [02miraheze/mw-config] 07MacFan4000 03bfb3822 - Update GlobalSettings.php [02:10:37] !log [@mwtask141] starting deploy of {'config': True} to all [02:10:38] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [02:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:44] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:10:45] !log [macfan@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [02:10:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:07] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [02:11:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:35] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 27s [02:11:38] miraheze/mw-config - MacFan4000 the build has errored. [02:11:39] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:11:40] !log [macfan@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [02:13:14] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bfb38224fb58...04f829d5e4d3 [02:13:16] [02miraheze/mw-config] 07MacFan4000 0304f829d - Update GlobalSettings.php [02:13:17] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [02:13:26] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 9s [02:13:28] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:13:34] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [02:14:18] miraheze/mw-config - MacFan4000 the build passed. [02:14:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:32] !log [@test131] starting deploy of {'config': True} to all [02:26:33] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:02:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.89, 6.08, 6.89 [03:03:05] PROBLEM - betaheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate '*.betaheze.org' expires in 7 day(s) (Fri 23 Jun 2023 18:56:37 GMT +0000). [03:03:40] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.357 second response time [03:03:56] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:04:24] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 21174 MB (15% inode=99%); [03:04:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.38, 7.61, 7.34 [03:05:28] PROBLEM - developer-and-testers.wiki.feign-wiki.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'developer-and-testers.wiki.feign-wiki.tk' expires in 15 day(s) (Sat 01 Jul 2023 18:48:08 GMT +0000). [03:05:29] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 10.90, 10.20, 4.41 [03:05:47] PROBLEM - landofliberos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [03:05:47] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.34, 4.73, 3.69 [03:05:53] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [03:05:59] PROBLEM - landofliberos.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - landofliberos.com reverse DNS resolves to any-in-2001-4860-4802-38--15.1e100.net [03:06:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.68, 3.21 [03:08:39] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.09, 3.94, 3.34 [03:10:19] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=decyclopediawiki (END - exit=0) [03:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.31, 3.75, 3.35 [03:10:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ccaf4497378f...662cd53c6046 [03:10:46] [02miraheze/ssl] 07MirahezeSSLBot 03662cd53 - Bot: Update SSL cert for developer-and-testers.wiki.feign-wiki.tk [03:11:22] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 0.42, 3.60, 3.24 [03:12:31] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.04, 3.20, 3.20 [03:12:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.71, 7.73, 7.61 [03:13:20] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.16, 2.45, 2.86 [03:16:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.46, 8.44, 7.94 [03:19:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.02, 3.61, 3.85 [03:20:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 7.87, 7.84 [03:22:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 3.72, 3.29 [03:24:11] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.52, 3.37, 3.22 [03:25:10] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.10, 2.50, 3.26 [03:26:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.77, 8.13, 7.92 [03:28:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.87, 3.41, 3.29 [03:30:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.51, 2.72, 3.05 [03:30:22] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.17, 2.92, 1.78 [03:31:15] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:33:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:33:52] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [03:34:08] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1013750732 secs [03:34:41] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:35:16] PROBLEM - mon141 Current Load on mon141 is WARNING: WARNING - load average: 3.21, 3.71, 2.42 [03:36:12] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:36:18] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 3.14, 2.50 [03:37:16] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.57, 2.54, 2.15 [03:39:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.17, 4.90, 3.78 [03:40:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.51, 3.33, 3.12 [03:42:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 2.52, 2.84 [03:45:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.70, 3.37, 3.51 [03:48:08] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09964221716 secs [03:48:37] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.13, 7.16, 6.56 [03:49:07] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [03:49:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.54, 4.07, 3.80 [03:50:33] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.35, 6.54, 6.40 [03:51:10] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [03:51:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.18, 3.69, 3.69 [03:52:27] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.49, 3.91, 3.24 [03:54:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.39, 3.33, 3.11 [03:55:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.88, 2.87, 3.36 [04:03:33] RECOVERY - developer-and-testers.wiki.feign-wiki.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'developer-and-testers.wiki.feign-wiki.tk' will expire on Thu 14 Sep 2023 02:10:38 GMT +0000. [04:03:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.91, 4.62, 3.84 [04:05:43] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:18:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 7.21, 7.97 [04:20:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.06, 8.05, 8.19 [04:27:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.50, 3.53, 3.92 [04:28:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.66, 7.77, 7.99 [04:29:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.57, 4.60, 2.35 [04:30:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 16.61, 10.66, 8.99 [04:33:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.82, 2.97, 2.24 [04:37:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.60, 4.23, 4.05 [04:41:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.83, 3.68, 3.87 [04:42:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.53, 2.92 [04:43:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.42, 3.87, 3.91 [04:44:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.38, 2.68, 2.68 [04:47:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.88, 3.61, 3.80 [04:53:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.45, 3.66, 3.72 [04:54:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.30, 4.52, 3.43 [04:55:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.80, 3.00, 3.48 [04:57:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.15, 3.12, 3.44 [04:58:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.36, 3.23 [04:59:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.89, 2.71, 3.25 [05:03:37] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:07:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.08, 7.34, 6.60 [05:09:22] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.95, 6.69, 6.44 [05:32:50] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:48:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 6.86, 7.82 [05:49:30] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [05:51:32] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [05:53:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:54:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.73, 7.62, 7.75 [05:55:26] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [05:58:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.30, 3.56, 2.93 [05:59:21] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wisrail.de. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:00:11] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 11.95, 6.43, 3.61 [06:00:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.63, 3.42, 2.97 [06:02:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.30, 3.97, 3.24 [06:06:06] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.84, 3.43, 3.27 [06:08:05] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.93, 2.63, 2.99 [06:10:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.69, 3.87, 3.69 [06:12:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.56, 7.34, 7.90 [06:16:07] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 4.39, 3.94 [06:16:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.64, 5.17, 3.33 [06:16:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.44, 8.01, 8.01 [06:18:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.12, 4.00, 3.86 [06:18:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.64, 3.62, 2.98 [06:23:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.34, 2.86, 3.40 [06:25:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.99, 4.19, 3.25 [06:27:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.01, 3.42, 3.08 [06:27:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.17, 3.22, 2.44 [06:27:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.95, 3.13, 3.43 [06:28:15] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wisrail.de All nameservers failed to answer the query. [06:29:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.96, 3.33, 3.12 [06:29:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 3.49, 3.52 [06:31:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.80, 3.92, 3.69 [06:32:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:33:57] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.13, 4.18, 3.80 [06:37:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.70, 3.48, 3.50 [06:39:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.30, 2.42, 3.11 [06:40:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.54, 6.27, 3.86 [06:42:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:42:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.62, 6.00, 4.09 [06:43:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.51, 3.53, 3.77 [06:49:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.94, 2.57, 3.23 [06:57:09] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: SSL OK - wiki.wisrail.de reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:01:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.62, 6.16, 3.75 [07:03:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:04:31] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:05:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.51, 3.84, 3.37 [07:06:29] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [07:07:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.97, 2.96, 3.10 [07:29:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.52, 5.13, 3.19 [07:32:44] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:35:07] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [07:37:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.25, 3.88, 3.60 [07:38:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.06, 7.92 [07:39:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.91, 2.86, 3.26 [07:46:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.03, 7.46 [08:03:31] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:04:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.10, 7.35, 7.77 [08:26:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.18, 7.19 [08:32:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.62, 7.10, 7.22 [08:33:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:40:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.07, 7.01 [08:42:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 7.01, 7.00 [08:44:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.65, 7.80, 7.27 [08:46:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.53, 7.24 [08:48:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 8.17, 7.50 [08:56:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.72, 7.53, 7.63 [09:06:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.14, 7.77, 7.48 [09:08:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.83, 7.55 [09:10:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.21, 7.96, 7.63 [09:16:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.38, 6.93, 7.32 [09:24:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.14, 7.18, 7.18 [09:26:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.49, 7.30 [09:28:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.30, 8.43, 7.65 [09:28:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.84, 3.62, 2.48 [09:30:51] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.52, 3.48, 2.56 [09:32:17] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:32:50] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.15, 2.60, 2.35 [09:34:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.68, 6.71, 6.18 [09:34:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.72, 7.63 [09:36:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.77, 6.43, 6.15 [09:44:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.35, 7.41 [09:48:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.91, 7.70, 7.56 [09:50:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 8.46, 7.89 [09:54:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 8.00, 7.83 [10:01:30] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:02:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.38, 8.85, 8.14 [10:16:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.61, 6.67, 7.63 [10:20:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.45, 8.49, 8.04 [10:28:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 7.85, 7.97 [10:32:40] PROBLEM - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.geoparkcorumbatai.com.br' expires in 15 day(s) (Sun 02 Jul 2023 10:30:17 GMT +0000). [10:38:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.55, 7.63, 7.67 [10:40:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.51, 5.78, 3.31 [10:40:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/662cd53c6046...c9849b952655 [10:40:42] [02miraheze/ssl] 07MirahezeSSLBot 03c9849b9 - Bot: Update SSL cert for wiki.geoparkcorumbatai.com.br [10:40:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.48, 7.61 [10:42:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.68, 4.58, 3.17 [10:42:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.40, 7.95, 7.77 [10:44:15] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.92, 6.44, 5.93 [10:46:11] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.60, 6.02, 5.83 [10:54:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.54, 7.06, 7.71 [10:58:49] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.38, 5.57, 3.13 [11:00:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.60, 7.73 [11:01:50] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:02:32] RECOVERY - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.geoparkcorumbatai.com.br' will expire on Thu 14 Sep 2023 09:40:33 GMT +0000. [11:02:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.55, 7.70 [11:06:44] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.52, 3.18, 3.02 [11:12:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.69, 7.71 [11:14:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.06, 7.37, 7.58 [11:20:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.83, 8.98, 8.11 [11:30:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.84, 7.12, 7.81 [11:34:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.35, 6.55, 5.93 [11:34:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.41, 7.72 [11:36:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.42, 6.83, 6.11 [11:36:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.55, 7.74 [11:38:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.97, 6.36, 6.02 [11:46:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.23, 8.64, 8.11 [11:52:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 7.91, 7.94 [11:56:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.95, 8.03, 7.94 [12:00:05] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.56, 4.59, 2.82 [12:00:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:02:03] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.66, 3.48, 2.62 [12:02:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 7.56, 7.81 [12:04:02] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.99, 2.64, 2.41 [12:04:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.85, 9.65, 8.53 [12:07:47] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:24:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.76, 7.27, 7.95 [12:26:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:30:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.74, 7.83, 7.77 [12:31:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:31:29] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.01, 6.77, 4.38 [12:35:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.56, 6.56, 4.91 [12:37:17] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:45:39] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.63, 3.52, 2.76 [12:47:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.83, 3.26, 2.75 [12:59:30] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.66, 4.01, 3.18 [13:00:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:01:29] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.89, 3.30, 3.02 [13:20:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:29:09] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.61, 4.07, 2.77 [13:31:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.43, 3.39, 2.68 [13:35:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.56, 2.79, 2.42 [13:37:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.43, 7.60, 6.91 [13:37:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.34, 2.75, 2.46 [13:41:01] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.32, 6.71, 6.68 [13:48:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:08:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.30, 5.73, 3.08 [14:10:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.90, 4.83, 3.09 [14:13:10] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.95, 3.11, 2.66 [14:13:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:10] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.56, 3.66, 2.92 [14:16:09] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:17:09] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.24, 3.14, 2.82 [14:18:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:20:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.45, 6.73, 6.40 [14:22:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.51, 6.24, 6.25 [14:27:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.88, 3.65, 2.96 [14:28:24] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.086 second response time [14:28:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 3.86, 3.12 [14:29:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.81, 4.71, 3.53 [14:30:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.01, 3.51, 3.10 [14:31:53] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.78, 3.60, 3.26 [14:32:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 2.93, 2.94 [14:35:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.99, 3.83, 3.38 [14:37:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.53, 2.91, 3.10 [14:50:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.69, 3.70, 3.08 [14:52:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.87, 3.45, 3.07 [14:54:43] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.56, 2.78, 2.87 [14:56:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:02:09] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [15:02:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.44, 7.08, 7.97 [15:04:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.63, 7.67, 8.05 [15:09:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.15, 3.51, 2.65 [15:11:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 3.05, 2.58 [15:22:47] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:39] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.01, 6.13, 5.31 [15:24:53] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.892 second response time [15:26:39] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.71, 5.83, 5.29 [15:28:17] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.44, 3.23, 2.77 [15:30:16] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.68, 2.79, 2.68 [15:36:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.95, 6.70, 6.57 [15:40:06] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.98, 6.50, 6.54 [15:48:49] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:27] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.27, 3.21, 2.30 [15:55:23] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:56:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:56:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.79, 2.72, 2.25 [15:57:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.93, 3.74, 2.81 [15:59:24] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.615 second response time [15:59:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.28, 3.28, 2.76 [16:01:01] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.19, 2.56, 1.34 [16:01:21] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:02:20] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.04, 3.57, 2.94 [16:02:44] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [16:03:01] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.33, 2.22, 1.38 [16:03:14] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [16:04:19] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.70, 3.25, 2.90 [16:06:44] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [16:08:16] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.74, 3.67, 3.18 [16:10:16] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.79, 3.25, 3.08 [16:11:24] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [16:13:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.52, 4.08, 3.06 [16:15:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.59, 3.49, 2.96 [16:16:05] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:17:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.49, 3.30, 2.96 [16:19:38] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.70, 3.41, 3.11 [16:21:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.13, 6.38, 4.03 [16:23:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.23, 2.65, 2.86 [16:28:01] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.908 second response time [16:30:05] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [16:35:11] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.45, 3.24, 2.39 [16:35:31] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.88, 2.91, 3.75 [16:36:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.66, 7.10, 6.61 [16:37:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.56, 2.56, 2.24 [16:37:29] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.60, 2.19, 3.39 [16:41:24] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:42:17] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.44, 6.31, 6.45 [16:45:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:46:54] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:48:52] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [16:53:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.04, 6.92, 6.65 [16:56:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.10, 6.59, 6.58 [16:59:41] PROBLEM - ambientguitar.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ambientguitar.org could not be found [17:05:30] PROBLEM - www.ambientguitar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.22369847616832:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1543:SSL alert number 40 [17:09:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.46, 3.67, 3.10 [17:10:54] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:13:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 2.93, 2.95 [17:16:20] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:19] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [17:19:54] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.40, 3.68, 2.67 [17:21:51] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.82, 3.99, 2.90 [17:22:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.06, 3.24, 2.68 [17:23:45] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.07, 7.01, 5.16 [17:24:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.79, 2.99, 2.65 [17:25:40] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.25, 5.84, 4.94 [17:25:47] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.68, 3.44, 2.95 [17:27:46] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.40, 3.91, 3.19 [17:29:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.90, 6.75, 6.58 [17:29:32] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.16, 7.20, 5.73 [17:29:42] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.52, 3.98, 3.31 [17:30:04] PROBLEM - ambient.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ambient.wiki could not be found [17:31:03] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.37, 6.12, 6.36 [17:31:40] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.52, 4.29, 3.52 [17:33:22] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.73, 8.01, 6.34 [17:35:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.95, 7.62, 6.38 [17:35:30] RECOVERY - www.ambientguitar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ambientguitar.org' will expire on Thu 14 Sep 2023 16:09:58 GMT +0000. [17:35:46] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:12] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.72, 8.21, 6.73 [17:39:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.597 second response time [17:40:48] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [17:42:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:58] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.74, 6.92, 6.77 [17:44:53] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.27, 5.76, 6.36 [17:45:10] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.32, 3.69, 3.92 [17:46:21] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.98, 2.93, 1.73 [17:47:09]