[00:02:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.14, 4.15, 3.53 [00:03:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.82, 7.07, 6.76 [00:04:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.50, 3.57, 3.40 [00:05:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.92, 7.62, 6.99 [00:06:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 3.19, 3.28 [00:06:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.81, 5.71, 3.20 [00:07:45] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.77, 3.05, 1.70 [00:08:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.77, 3.54, 3.25 [00:08:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.98, 5.35, 3.38 [00:09:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 7.71, 7.22 [00:09:48] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.83, 2.90, 1.82 [00:10:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.02, 3.62, 3.48 [00:12:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.00, 3.00, 3.27 [00:12:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.50, 6.31, 4.22 [00:14:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.29, 2.95, 3.10 [00:14:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.96, 7.76, 5.02 [00:16:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.80, 6.49, 4.88 [00:20:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.25, 4.45, 3.65 [00:20:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.84, 9.03, 7.87 [00:21:34] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:22:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.88, 3.94, 3.55 [00:24:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.65, 4.77, 3.91 [00:28:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.86, 3.72 [00:28:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.23, 7.77, 7.70 [00:30:19] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.75, 3.35, 2.62 [00:30:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 8.09, 7.84 [00:30:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.87, 8.86, 6.40 [00:32:15] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.92, 3.20, 2.65 [00:34:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.98, 7.89 [00:35:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:36:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.13, 7.52, 6.73 [00:38:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.97, 2.97, 3.35 [00:38:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.22, 7.38, 6.75 [00:40:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.41, 7.25, 6.77 [00:42:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.74, 7.76 [00:42:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.16, 6.13, 6.42 [00:44:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 7.30, 7.59 [00:45:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:48:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.50, 8.24, 7.24 [00:49:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.98, 7.98, 7.76 [00:50:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.89, 7.97, 7.28 [00:53:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 7.93, 7.85 [00:54:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.58, 5.98, 6.66 [00:59:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.15, 7.77, 7.69 [01:01:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.55, 7.61 [01:03:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 7.66, 7.62 [01:05:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.98, 3.54 [01:05:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.06, 7.33, 7.50 [01:08:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.36, 7.58, 6.57 [01:08:36] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.96, 3.53, 3.01 [01:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.19, 7.53, 7.51 [01:10:33] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.95, 2.93, 2.85 [01:10:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.79, 5.80, 6.05 [01:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 7.44, 7.50 [01:13:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.12, 4.60, 3.83 [01:19:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.54, 3.59, 3.70 [01:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.43, 8.18, 7.78 [01:25:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.63, 4.48, 3.94 [01:25:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 7.69, 7.65 [01:27:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.32, 3.93, 3.80 [01:31:02] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [01:31:04] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/7422089ade78...3c801481720f [01:31:07] [02miraheze/ManageWiki] 07AgentIsai 033c80148 - Fix [01:31:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.78, 4.26, 3.91 [01:33:16] miraheze/ManageWiki - AgentIsai the build has errored. [01:33:39] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [01:33:41] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/3c801481720f...c659c5120361 [01:33:42] [02miraheze/ManageWiki] 07AgentIsai 03c659c51 - Missing semicolon [01:34:56] miraheze/ManageWiki - AgentIsai the build has errored. [01:35:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.33, 3.58, 3.70 [01:38:26] miraheze/ManageWiki - AgentIsai the build has errored. [01:41:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.61, 2.85, 3.38 [01:42:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.96, 3.92, 3.28 [01:43:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.39, 8.25, 7.73 [01:44:28] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.12, 3.51, 3.20 [01:45:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.22, 3.82, 3.60 [01:46:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.98, 2.99, 3.05 [01:47:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.27, 3.28, 3.44 [01:49:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.24, 4.89, 3.99 [01:53:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.58, 3.66 [01:53:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.90, 7.98 [01:55:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.60, 8.28, 8.11 [01:57:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.52, 3.20 [02:03:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.50, 5.36, 4.12 [02:07:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.92, 3.54, 3.23 [02:08:38] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/c659c5120361...0db4ce60aa45 [02:08:39] [02miraheze/ManageWiki] 07AgentIsai 030db4ce6 - Fix variable [02:08:42] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:09:18] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.01, 2.97, 3.06 [02:09:50] [02miraheze/ManageWiki] 07github-actions[bot] pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/0db4ce60aa45...a7f34e421237 [02:09:52] [02miraheze/ManageWiki] 07github-actions 03a7f34e4 - CI: lint code to MediaWiki standards [02:09:53] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:10:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.95, 5.34, 3.13 [02:12:00] miraheze/ManageWiki - AgentIsai the build passed. [02:13:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.12, 3.41, 3.18 [02:14:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.66, 4.96, 3.57 [02:15:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.76, 3.11, 3.09 [02:15:09] !log [www-data@test131] starting deploy of {'folders': 'extensions/ManageWiki'} to all [02:15:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:16] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [02:15:17] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [02:15:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:43] ^ me [02:17:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.04, 3.37, 3.90 [02:23:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.51, 2.38, 3.31 [02:24:53] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.68, 5.04, 3.90 [02:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 7.60, 7.98 [02:28:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.59, 3.58 [02:32:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.97, 2.84, 3.28 [02:35:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.73, 3.85, 3.34 [02:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 14.22, 9.30, 8.30 [02:36:36] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.46, 4.64, 3.87 [02:37:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.01, 3.47, 3.26 [02:38:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.90, 3.79, 3.64 [02:39:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.90, 3.37, 3.25 [02:43:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.82, 4.22, 3.73 [02:45:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.36, 3.64, 3.57 [02:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.09, 7.61, 7.95 [02:46:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 2.98, 3.32 [02:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 8.15, 8.09 [02:53:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.01, 3.70, 3.65 [02:53:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.82, 7.21, 7.76 [02:59:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.77, 3.86, 3.84 [03:03:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.78, 3.40 [03:03:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.83, 7.44, 7.54 [03:04:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 5.03, 4.00 [03:05:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.15, 7.43, 7.53 [03:09:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.60, 5.23, 4.14 [03:11:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.60, 3.99, 3.82 [03:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.50, 7.49 [03:12:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.07, 3.33, 3.64 [03:13:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.62, 7.54 [03:14:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.72, 2.77, 3.40 [03:17:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.16, 4.10, 3.80 [03:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.07, 9.36, 8.16 [03:19:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.76, 3.63, 3.65 [03:23:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.87, 4.41, 3.93 [03:23:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.72, 3.55, 3.34 [03:25:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.42, 3.57, 3.68 [03:25:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.78, 3.56, 3.37 [03:27:43] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.15, 3.08, 3.22 [03:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 7.44, 7.99 [03:31:24] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hotararewiki --no-updates --username-prefix=w /home/agent/ImportDump/hotararewiki-20230513024153.xml (START) [03:31:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.62, 2.54, 3.21 [03:33:17] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hotararewiki --no-updates --username-prefix=w /home/agent/ImportDump/hotararewiki-20230513024153.xml (END - exit=0) [03:33:18] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hotararewiki (START) [03:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.21, 7.40, 7.73 [03:36:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.06, 4.93, 2.60 [03:40:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.71, 7.18, 4.13 [03:41:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.50, 3.71, 3.44 [03:42:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.77, 5.06, 3.71 [03:42:56] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 10.18, 5.08, 3.06 [03:43:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.13, 3.14, 3.27 [03:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.28, 7.40, 7.74 [03:47:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.05, 3.47, 3.38 [03:49:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.84, 3.97, 3.57 [03:55:06] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.55, 4.12, 3.18 [03:57:04] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.76, 3.26, 2.97 [04:01:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.23, 3.65, 3.98 [04:02:13] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [04:03:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 6.79, 7.06 [04:05:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.84, 6.76, 7.04 [04:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.19, 7.11 [04:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.69, 6.56, 6.89 [04:13:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.22, 2.95, 3.39 [04:13:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.31, 6.24, 6.74 [04:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.10, 7.01 [04:19:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.94, 7.46, 7.13 [04:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.41, 6.98, 7.01 [04:30:58] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:31:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.10, 6.15, 6.67 [04:31:57] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hotararewiki (END - exit=0) [04:31:58] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=hotararewiki --active --update (END - exit=0) [04:32:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:32:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:34:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.92, 3.28, 3.13 [04:36:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.22, 4.10, 3.45 [04:40:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.22, 3.37, 3.32 [04:42:11] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [04:42:13] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/a7f34e421237...ed38a5d5f1ab [04:42:15] [02miraheze/ManageWiki] 07AgentIsai 03ed38a5d - Fix breadcrumb navigation [04:42:27] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [04:42:28] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [04:42:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.70, 7.97, 7.17 [04:42:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:45:00] miraheze/ManageWiki - AgentIsai the build has errored. [04:46:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 7.48, 7.15 [04:52:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.63, 7.27 [04:53:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.54, 4.08, 3.08 [04:54:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 7.39, 7.23 [04:58:56] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.89, 1.97, 3.88 [04:59:42] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.31, 3.62, 3.27 [05:01:20] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/ed38a5d5f1ab...d039a34ec755 [05:01:21] [02miraheze/ManageWiki] 07AgentIsai 03d039a34 - Change variable names [05:01:24] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [05:01:40] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.95, 4.16, 3.51 [05:01:41] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [05:01:42] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [05:01:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:01:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.81, 7.48 [05:02:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.38, 1.72, 3.34 [05:03:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.91, 3.72, 3.43 [05:03:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.46, 3.04, 2.71 [05:04:06] miraheze/ManageWiki - AgentIsai the build has errored. [05:05:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.74, 3.19, 2.82 [05:07:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.59, 3.23, 3.30 [05:09:07] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [05:09:10] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/d039a34ec755...d2b047de8c51 [05:09:12] [02miraheze/ManageWiki] 07AgentIsai 03d2b047d - Fix parameters [05:11:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.09, 7.99, 7.96 [05:12:13] miraheze/ManageWiki - AgentIsai the build passed. [05:13:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.51, 5.10, 4.07 [05:19:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.58, 8.52, 8.06 [05:21:17] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.90, 3.89, 3.87 [05:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 7.88, 7.95 [05:23:51] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:29:07] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.44, 3.23, 3.51 [05:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.20, 7.97, 7.93 [05:31:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.95, 3.34, 3.51 [05:32:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.46, 4.51, 3.48 [05:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 7.56, 7.84 [05:38:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.10, 2.90, 3.31 [05:41:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.03, 8.37, 7.99 [05:42:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.39, 3.85, 3.91 [05:43:49] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/d2b047de8c51...c1d81203f52c [05:43:50] [02miraheze/ManageWiki] 07AgentIsai 03c1d8120 - i18n [05:43:51] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [05:44:15] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [05:44:16] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 1s [05:44:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:22] !log [www-data@test131] starting deploy of {'l10n': True, 'folders': 'w/extensions/ManageWiki'} to all [05:44:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.93, 7.93 [05:47:03] miraheze/ManageWiki - AgentIsai the build passed. [05:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.29, 8.70, 8.18 [05:50:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.81, 2.48, 3.24 [05:52:12] [02puppet] 07Reception123 closed pull request 03#3214: deploy-mediawiki: use --extensions-dir for mergeMessageFileList.php - 13https://github.com/miraheze/puppet/pull/3214 [05:52:14] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/764cc44bf91d...9aa97ba97a6e [05:52:16] [02miraheze/puppet] 07Universal-Omega 039aa97ba - deploy-mediawiki: use --extensions-dir for mergeMessageFileList.php (#3214) [05:52:18] [02mw-config] 07Reception123 closed pull request 03#5229: Remove most of extension-list - 13https://github.com/miraheze/mw-config/pull/5229 [05:52:21] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/abde38285813...e812ac132149 [05:52:24] [02miraheze/mw-config] 07Universal-Omega 03e812ac1 - Remove most of extension-list (#5229) [05:53:18] miraheze/mw-config - Reception123 the build passed. [05:53:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 4.00, 3.76, 3.37 [05:53:34] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [05:55:12] !log [www-data@test131] finished deploy of {'l10n': True, 'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 649s [05:55:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:30] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 3.40, 3.31 [05:58:13] !log [@test131] starting deploy of {'config': True} to all [05:58:14] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:58:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:05:15] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:10:51] !log [@mwtask141] starting deploy of {'config': True} to all [06:10:56] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.66, 3.35, 2.40 [06:11:08] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 17s [06:11:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.37, 3.25, 2.48 [06:14:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.84, 3.66, 3.28 [06:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.54, 7.05, 7.83 [06:19:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.61, 7.95 [06:23:17] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.85, 7.50, 7.99 [06:27:49] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.26, 3.63, 1.91 [06:28:34] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.62, 6.39, 3.69 [06:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.92, 7.96, 8.07 [06:29:49] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.85, 3.56, 2.11 [06:30:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.74, 6.38, 4.02 [06:31:49] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.11, 2.69, 1.97 [06:32:56] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.51, 3.07 [06:33:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.52, 7.91 [06:34:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.95, 3.27, 3.04 [06:37:45] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.54, 2.48, 1.62 [06:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 7.65, 7.84 [06:38:44] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2319024503 secs [06:39:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.20, 2.34, 3.88 [06:39:45] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.14, 2.06, 1.58 [06:39:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 7.42, 7.74 [06:45:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.18, 2.16, 3.35 [06:48:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.67, 11.52, 11.98 [06:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.06, 7.48, 7.50 [06:50:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.89, 11.85, 12.06 [06:50:44] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.008927792311 secs [06:51:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 6.95, 7.30 [06:52:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.22, 11.34, 11.83 [06:53:00] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:57:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.44, 7.73, 7.43 [07:01:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.23, 7.95, 7.62 [07:03:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:04:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.00, 11.04, 10.99 [07:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.32, 10.98, 10.96 [07:08:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.05, 11.91, 11.29 [07:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 7.70, 7.52 [07:15:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.68, 7.43, 7.47 [07:19:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 8.05, 7.68 [07:20:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.10, 11.93, 11.90 [07:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 7.80, 7.64 [07:22:43] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:23:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.17, 10.70, 11.92 [07:24:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.99, 13.30, 12.35 [07:25:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.33, 13.03, 12.63 [07:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.63, 7.75, 7.54 [07:27:47] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.51, 7.47 [07:33:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.83, 10.25, 11.72 [07:33:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.71, 7.44, 7.38 [07:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.64, 7.37, 7.35 [07:38:01] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.667 second response time [07:41:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.34, 12.56, 11.95 [07:41:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.61, 8.18, 7.52 [07:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.99, 7.63 [07:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.69, 8.41, 7.88 [08:01:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 11.13, 11.91 [08:05:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 2.59, 1.76 [08:07:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.56, 11.57, 11.68 [08:07:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.03, 2.60, 1.86 [08:16:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.95, 9.58, 11.57 [08:17:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 11.39, 11.95 [08:19:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.76, 11.73, 12.02 [08:21:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.25, 11.24, 11.81 [08:22:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:30:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 8.77, 10.17 [08:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.45, 11.50, 10.98 [08:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 10.60, 10.73 [08:37:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.19, 11.86, 11.63 [08:38:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.98, 11.45, 11.02 [08:48:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 11.21, 11.37 [08:58:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 10.67, 10.92 [09:00:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.60, 9.61, 10.51 [09:01:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.93, 10.71, 11.79 [09:02:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.22, 8.98, 10.18 [09:05:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.65, 7.25, 7.94 [09:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.38, 9.90, 10.30 [09:11:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.04, 10.74, 10.86 [09:13:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.82, 10.82, 10.86 [09:13:58] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1071780026 secs [09:18:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.15, 9.57, 10.08 [09:25:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.35, 9.47, 10.18 [09:29:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.88, 10.57, 10.50 [09:29:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.74, 6.00, 6.70 [09:33:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 10.89, 10.60 [09:35:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.10, 11.14, 10.42 [09:35:57] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.0960714519 secs [09:38:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.13, 10.04, 10.14 [09:39:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.47, 10.62, 10.67 [09:41:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.03, 11.63, 11.03 [09:43:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 11.03, 10.90 [09:48:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.75, 6.33 [09:52:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.92, 6.54, 6.36 [09:57:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.29, 10.66, 10.62 [09:59:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.21, 10.52, 10.58 [10:03:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.43, 9.66, 10.19 [10:06:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.50, 11.18, 9.85 [10:08:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 10.35, 9.70 [10:10:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.60, 11.81, 10.30 [10:11:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.29, 12.56, 11.17 [10:12:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 11.24, 10.26 [10:16:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.47, 11.37, 10.50 [10:21:58] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1240583956 secs [10:22:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.80, 11.93, 11.05 [10:23:56] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:26:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.76, 11.87, 11.18 [10:28:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.78, 10.33, 10.70 [10:30:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.29, 11.32, 11.03 [10:32:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.76, 11.26, 11.06 [10:34:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.19, 12.12, 11.38 [10:36:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.71, 11.20, 11.14 [10:39:57] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09714666009 secs [10:43:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.69, 11.42, 11.93 [10:48:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.44, 9.05, 10.03 [10:53:38] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [10:55:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.17, 10.35, 10.78 [10:58:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.69, 10.77, 10.24 [11:00:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 10.42, 10.18 [11:01:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.62, 11.72, 11.30 [11:02:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.13, 11.27, 10.51 [11:03:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.75, 6.79 [11:06:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.55, 11.70, 10.91 [11:07:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 11.59, 11.33 [11:07:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.48, 6.67, 6.54 [11:07:58] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset -0.107698977 secs [11:08:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 12.02, 11.13 [11:14:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.07, 11.20, 11.11 [11:15:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.73, 6.85, 6.70 [11:17:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 11.48, 11.87 [11:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.98, 7.39, 6.91 [11:17:57] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.09820654988 secs [11:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.70, 7.43, 7.10 [11:22:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.31, 9.26, 10.18 [11:29:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.58, 11.45, 11.28 [11:29:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.80, 6.40, 6.72 [11:33:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.64, 11.34, 11.32 [11:41:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.24, 12.39, 11.56 [11:45:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.59, 11.88, 11.59 [11:47:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 7.04, 6.86 [11:49:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.57, 6.71 [11:53:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.02, 10.91, 10.05 [11:55:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 10.41, 9.97 [11:57:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.56, 12.71, 11.78 [11:59:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 11.11, 10.32 [12:01:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.06, 11.39, 10.55 [12:04:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.96, 12.31, 11.03 [12:10:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.18, 11.06, 10.97 [12:10:56] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 2.92, 2.18 [12:12:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.39, 2.75, 2.21 [12:16:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.53, 11.46, 11.10 [12:18:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.83, 11.26, 11.08 [12:18:37] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:20:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.13, 12.50, 11.59 [12:22:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.31, 6.79 [12:22:47] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:24:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.00, 6.59, 6.60 [12:25:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.49, 3.73, 2.67 [12:27:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 3.04, 2.54 [12:40:33] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.39, 3.10 [12:44:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.66, 3.21, 3.11 [12:47:12] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:00:44] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2491616011 secs [13:07:30] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.32, 3.63, 3.27 [13:13:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.84, 3.96, 3.54 [13:17:23] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 9.35, 5.18, 2.67 [13:19:02] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.71, 2.91, 3.24 [13:21:23] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.93, 2.99, 2.36 [13:26:44] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.04405748844 secs [13:32:34] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.46, 5.70, 3.15 [13:34:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.51, 4.06, 2.86 [13:36:29] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.62, 3.33, 2.63 [13:36:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.42, 6.58 [13:38:29] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.79, 3.32, 2.70 [13:40:29] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.53, 3.65, 2.89 [13:40:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.60, 6.83 [13:42:29] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.47, 3.48, 2.92 [13:44:29] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.70, 3.23, 2.90 [13:44:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.97, 7.03, 6.80 [13:46:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.14, 7.21, 6.87 [13:48:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.62, 7.21, 6.91 [13:54:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.25, 7.60, 7.08 [13:56:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.67, 7.83, 7.25 [13:58:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.00, 8.41, 7.54 [14:00:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.36, 7.70, 7.40 [14:08:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:09:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.24, 4.45, 3.05 [14:15:34] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.90, 3.24 [14:17:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.29, 3.69, 3.22 [14:18:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:19:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.25, 3.18, 3.09 [14:27:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 5.96, 6.70 [14:31:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.03, 6.84, 6.94 [14:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.90, 7.87, 7.31 [14:39:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 7.34, 7.19 [14:40:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.29, 3.89, 3.07 [14:42:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.49, 3.00, 2.84 [14:43:46] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:47] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.792 second response time [14:47:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:47:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.32, 6.75 [14:48:11] PROBLEM - kalons-reverie.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kalons-reverie.com' expires in 15 day(s) (Mon 29 May 2023 14:37:43 GMT +0000). [14:52:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:54:00] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1155606806 secs [14:54:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6349769dceb9...39f58991d011 [14:54:28] [02miraheze/ssl] 07MirahezeSSLBot 0339f5899 - Bot: Update SSL cert for kalons-reverie.com [15:01:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.40, 6.99 [15:02:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 2.87, 2.47 [15:03:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 7.00, 6.89 [15:04:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 2.36, 2.33 [15:05:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.57, 6.74 [15:09:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.083 second response time [15:20:47] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.82, 3.17, 2.78 [15:24:45] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.02, 3.32, 2.94 [15:26:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.71, 7.06 [15:28:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.85, 7.62, 7.07 [15:32:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.63, 7.19 [15:35:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9aa97ba97a6e...f53a9b5dcba2 [15:35:32] [02miraheze/puppet] 07paladox 03f53a9b5 - matomo: update to 4.14.2 [15:36:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 7.95, 7.36 [15:37:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.71, 3.41, 2.80 [15:38:44] !log upgrade grafana on mon141 [15:38:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:39:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.29, 3.34, 2.87 [15:41:46] !log mon141: apt-get dist-upgrade & reboot [15:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:20] !log puppet141: apt-get dist-upgrade & reboot [15:42:35] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 19.76, 11.14, 5.62 [15:42:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:52] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.42, 3.56, 1.89 [15:43:59] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.75, 6.97, 7.30 [15:45:50] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.33, 2.74, 1.79 [15:45:55] !log ldap141: apt-get dist-upgrade & reboot [15:45:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.866 second response time [15:46:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:46:02] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.28, 3.27, 2.16 [15:47:11] RECOVERY - puppet141 APT on puppet141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:47:23] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.35, 3.79, 2.04 [15:47:33] RECOVERY - kalons-reverie.com - LetsEncrypt on sslhost is OK: OK - Certificate 'kalons-reverie.com' will expire on Fri 11 Aug 2023 13:54:16 GMT +0000. [15:47:59] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 2.65, 3.11, 2.24 [15:49:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f53a9b5dcba2...3c2962799a75 [15:49:27] [02miraheze/puppet] 07paladox 033c29627 - Fix miraheze-backup script [15:49:33] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:49:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.19, 7.23 [15:50:19] RECOVERY - ldap141 APT on ldap141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:51:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3c2962799a75...d5a5cd9c5a8b [15:51:07] [02miraheze/puppet] 07paladox 03d5a5cd9 - Fix typo in miraheze-backup [15:51:08] PROBLEM - ldap141 Current Load on ldap141 is CRITICAL: CRITICAL - load average: 2.10, 1.30, 0.61 [15:51:49] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.18, 3.66, 2.65 [15:51:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.22, 7.22 [15:55:28] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 4.37, 1.03, 0.34 [15:55:55] PROBLEM - puppet141 puppetserver on puppet141 is CRITICAL: connect to address 2a10:6740::6:507 and port 8140: Connection refused [15:55:56] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:55:57] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09435123205 secs [15:56:10] PROBLEM - puppet141 puppetdb on puppet141 is CRITICAL: connect to address 2a10:6740::6:507 and port 8081: Connection refused [15:56:56] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:57:26] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:57:26] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 1.09, 0.94, 0.39 [15:57:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:57:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.43, 3.15 [15:57:50] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:58:06] RECOVERY - puppet141 puppetdb on puppet141 is OK: TCP OK - 0.000 second response time on 2a10:6740::6:507 port 8081 [15:58:15] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.47, 6.30, 6.74 [15:58:46] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:59:24] PROBLEM - cp33 Puppet on cp33 is CRITICAL: CRITICAL: Puppet has 929 failures. Last run 2 minutes ago with 929 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/stablestate.org.key],File[/etc/ssl/localcerts/wiki.joust.ro.crt],File[/etc/ssl/private/wiki.joust.ro.key],File[/etc/ssl/localcerts/en.gyaanipedia.com.crt] [15:59:34] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.30, 4.21, 3.47 [15:59:55] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:56] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:00:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d5a5cd9c5a8b...3c0bfdf2cf6d [16:00:37] [02miraheze/puppet] 07paladox 033c0bfdf - Fix [16:01:12] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:01:51] RECOVERY - puppet141 puppetserver on puppet141 is OK: TCP OK - 0.000 second response time on 2a10:6740::6:507 port 8140 [16:03:09] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:03:09] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:03:16] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:03:37] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:04:30] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:05:06] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:05:39] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:05:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.62, 8.52, 7.67 [16:07:14] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:09:44] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:11:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 7.98, 7.71 [16:12:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.41, 3.29, 3.82 [16:13:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.55, 8.65, 7.96 [16:14:28] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.208 second response time [16:17:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.81, 7.36, 7.62 [16:21:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.62, 7.65 [16:23:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.69, 3.35 [16:23:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.95, 7.04, 7.43 [16:24:54] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [16:24:56] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:25:20] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:25:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.23, 7.43 [16:27:22] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:28:10] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:28:51] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:29:08] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:29:08] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:29:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.15, 7.31, 7.41 [16:30:49] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:05] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:18] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:32:28] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:24] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:36:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:37:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.92, 2.68, 2.43 [16:37:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.07, 6.79 [16:39:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/39f58991d011...c45ab8448da8 [16:39:25] [02miraheze/ssl] 07MirahezeSSLBot 03c45ab84 - Bot: Add SSL cert for www.hebammenwiki.de [16:39:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.87, 2.80, 2.51 [16:40:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c45ab8448da8...353e3f0940c2 [16:40:47] [02miraheze/ssl] 07MirahezeSSLBot 03353e3f0 - Bot: Add SSL cert for bushcraftwiki.com [16:51:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.86, 7.13, 6.87 [17:01:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.57, 7.18 [17:03:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 7.09, 7.05 [17:06:00] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.97, 3.68, 2.88 [17:07:58] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.42, 3.43, 2.87 [17:09:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.81, 6.31, 6.80 [17:09:56] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.69, 3.09, 2.81 [17:13:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.36, 7.14 [17:20:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.18, 4.28, 3.29 [17:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.75, 8.13, 7.55 [17:23:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.65, 7.44 [17:24:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 3.94, 3.36 [17:26:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.58, 5.21, 3.90 [17:27:42] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [17:29:58] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.15, 3.54, 3.50 [17:31:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.31, 7.59, 7.44 [17:33:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.57, 7.45 [17:35:51] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 2.89, 3.29 [17:43:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.73, 8.49, 7.78 [17:44:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.42, 4.25, 3.21 [17:47:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.72, 7.65 [17:48:03] paladox: restart MirahezeRC [17:48:09] It's not listening right [17:48:16] Think it might have split brained [17:48:17] I’m mobile [17:48:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.77, 3.81, 3.37 [17:48:47] @Site Reliability Engineers [17:50:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.23, 3.20 [17:51:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.42, 7.51 [17:56:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.13, 4.47, 3.45 [17:57:25] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [17:58:20] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.91, 4.11, 2.27 [17:58:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.70, 3.92, 3.36 [18:00:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.02, 3.35, 2.22 [18:00:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.60, 3.17, 3.16 [18:02:38] hmm, yeah seems it failed auth and therefore cannot say anything in -feed [18:03:22] @Reception123 or @Orduin can we please restart ircrcbot? (mon141) [18:04:56] Or CosmicAlpha can voice it [18:07:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.49, 7.70, 8.00 [18:10:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.44, 3.43, 2.53 [18:11:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.46, 7.78 [18:12:51] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.48, 2.66 [18:13:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.57, 6.36, 7.34 [18:14:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.77, 3.21, 2.66 [18:15:36] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1687001586 secs [18:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.27, 8.58, 7.87 [18:23:35] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.05352041125 secs [18:25:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.13, 7.86, 7.81 [18:27:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.12, 8.02, 7.86 [18:33:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.47, 7.89, 7.90 [18:36:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.13, 4.22, 3.08 [18:38:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.02, 3.66, 3.01 [18:39:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.22, 7.55 [18:40:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 2.98, 2.84 [18:41:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.89, 6.63, 7.29 [18:41:11] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.40, 3.00 [18:43:08] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.85, 3.25, 3.00 [18:47:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.21, 6.02, 6.80 [18:51:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.22, 3.74, 2.95 [18:55:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.85, 3.87, 3.26 [18:57:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.82, 4.05, 3.35 [18:57:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.95, 6.95 [18:58:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.22, 3.33, 1.92 [18:59:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.80, 3.49, 3.23 [18:59:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.88, 7.57, 7.15 [19:00:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.15, 2.45, 1.77 [19:00:59] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.17, 3.12, 3.13 [19:03:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.54, 7.25 [19:05:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.49, 7.24 [19:05:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.49, 5.26, 3.52 [19:06:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.69, 3.99, 3.47 [19:07:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.11, 7.13 [19:08:47] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.61, 3.44, 3.33 [19:10:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.33, 2.94, 3.15 [19:13:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 3.36, 3.32 [19:13:45] !log [void@mon141] restart ircrcbot [19:13:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.71, 6.94, 6.99 [19:17:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 6.54, 6.83 [19:25:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.95, 3.71, 3.22 [19:27:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.05, 6.36, 6.72 [19:27:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 3.00, 3.01 [19:30:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.48, 3.58, 2.29 [19:31:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.21, 7.40, 7.03 [19:32:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 2.61, 2.09 [19:33:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 7.23, 7.02 [19:35:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.52, 6.16, 6.65 [19:37:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.93, 3.99, 3.42 [19:39:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.95, 3.20, 3.20 [19:45:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.31, 6.85 [19:47:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 6.86, 6.74 [19:49:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.67, 7.05 [19:51:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.51, 7.08 [19:51:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2533266544 secs [19:57:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.42, 6.71 [19:59:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.16, 3.53, 3.08 [20:01:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.11, 2.98 [20:01:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 7.76, 7.16 [20:03:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.57, 7.15 [20:11:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.42, 6.78 [20:19:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.17, 6.95, 6.88 [20:19:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.31, 3.41, 3.27 [20:21:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.04324412346 secs [20:21:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.62, 5.21, 3.90 [20:25:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.98, 6.57, 6.79 [20:27:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.78, 3.17, 3.40 [20:30:52] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:31:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.35, 3.42 [20:33:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.02, 6.76, 6.83 [20:33:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.97, 3.73, 3.52 [20:34:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.41, 4.27, 3.25 [20:35:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.34, 3.15, 3.33 [20:37:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.08, 3.42, 3.18 [20:39:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 7.08, 6.94 [20:39:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.55, 3.17, 3.11 [20:41:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.90, 6.74, 6.82 [20:43:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.10, 7.06, 6.91 [20:45:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.53, 6.73 [20:45:44] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.29, 3.52, 3.22 [20:47:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.93, 3.18, 3.13 [21:12:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.95, 6.91, 6.35 [21:12:42] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.24, 4.23, 3.33 [21:14:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.61, 6.31 [21:14:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.05, 3.65, 3.22 [21:16:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.47, 3.28, 3.13 [21:20:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.73, 7.95, 6.87 [21:21:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.08, 7.28, 6.76 [21:22:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.85, 3.60, 3.44 [21:23:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.13, 6.56, 6.56 [21:24:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.42, 3.37, 3.38 [21:30:18] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [21:38:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.15, 6.70 [21:40:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.31, 7.17, 6.78 [21:42:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.46, 6.55, 6.60 [21:46:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 6.99, 6.82 [21:48:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.15, 6.69, 6.73 [21:51:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.52, 7.09 [21:55:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.77, 6.61, 6.80 [22:19:07] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.12, 3.97, 3.20 [22:21:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.53, 3.49, 3.10 [22:21:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.01, 6.80 [22:26:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.75, 2.90, 2.99 [22:27:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.91, 6.33, 6.57 [22:31:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.71, 8.53, 7.42 [22:33:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.46, 3.32, 2.87 [22:35:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.77, 3.99, 3.17 [22:35:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.02, 7.59, 7.33 [22:35:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.25, 3.47, 3.21 [22:37:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.97, 3.58, 3.12 [22:39:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.01, 2.99, 2.96 [22:39:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.08, 3.15 [22:41:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.34, 7.69, 7.39 [22:43:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 7.27, 7.29 [22:51:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.03, 7.81, 7.42 [22:53:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.15, 7.04, 7.18 [22:54:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 16.72, 8.98, 5.25 [22:57:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.07, 4.03, 3.38 [22:57:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.09, 6.75 [22:59:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 3.39, 3.21 [23:03:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.72, 4.28, 3.66 [23:04:11] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.07, 3.20, 3.81 [23:08:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.54, 2.46, 3.39 [23:09:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.47, 3.17, 3.43 [23:11:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 2.68, 3.21 [23:25:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.50, 6.84, 6.48 [23:27:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.11, 3.05 [23:27:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.49, 6.65, 6.45 [23:29:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.05, 3.35, 3.13 [23:31:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.97, 3.43, 3.18 [23:31:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 6.88, 6.61 [23:33:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 3.05, 3.07 [23:33:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.24, 6.27, 6.41 [23:40:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.83, 10.71, 11.94 [23:44:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.64, 7.04, 6.78 [23:46:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.01, 6.76, 6.71 [23:48:12] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1282654703 secs [23:48:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.26, 11.18, 11.46 [23:57:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 6.95, 6.85 [23:59:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.52, 6.55, 6.70