[00:02:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.63, 6.63, 4.16 [00:02:12] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.005367398262 secs [00:02:22] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.76, 3.31, 2.35 [00:03:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 17.18, 10.85, 8.31 [00:03:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.62, 6.47, 4.66 [00:04:22] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.05, 3.22, 2.44 [00:09:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.51, 7.64, 7.71 [00:09:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.06, 3.59, 3.69 [00:13:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.36, 3.75, 3.66 [00:15:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.50, 3.21, 3.47 [00:17:42] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.70, 4.10, 3.76 [00:19:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.43, 2.94, 3.72 [00:21:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.88, 3.98, 3.85 [00:23:35] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.18, 4.72, 4.12 [00:25:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 2.72, 3.36 [00:25:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.76, 3.97, 3.91 [00:27:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.14, 6.76 [00:29:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.24, 2.51, 3.32 [00:29:45] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.97, 6.34, 4.20 [00:31:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.47, 6.32, 4.47 [00:33:22] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.74, 3.54, 3.60 [00:35:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.44, 7.30, 6.94 [00:35:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.65, 2.83, 3.33 [00:37:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.72, 3.77, 3.35 [00:37:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.77, 6.80 [00:39:41] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.45, 3.70, 2.94 [00:41:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.45, 7.09 [00:41:41] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.34, 3.64, 3.01 [00:43:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 7.32, 7.09 [00:43:41] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.36, 3.15, 2.91 [00:45:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.14, 6.46, 6.80 [00:49:37] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.86, 6.95, 5.82 [00:52:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 6.77, 6.85 [00:53:35] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.87, 7.88, 6.54 [00:53:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.58, 6.00, 6.56 [00:57:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.22, 3.24, 3.95 [00:57:34] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.39, 6.07, 6.13 [01:03:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.61, 3.96, 3.29 [01:07:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.96, 2.56, 3.32 [01:07:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.53, 3.08, 3.06 [01:15:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 6.59, 6.54 [01:15:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.03, 3.41, 3.15 [01:17:26] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.54, 3.09, 3.07 [01:19:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 6.99, 6.67 [01:21:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.19, 6.69, 6.60 [01:33:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.50, 6.88 [01:35:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.61, 6.75, 6.68 [01:38:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.95, 4.27, 3.66 [01:40:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.31, 3.36, 3.40 [01:51:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.80, 7.02, 6.86 [01:53:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.71, 6.77 [01:56:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.63, 3.82, 3.33 [02:00:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.35, 4.07, 3.52 [02:02:09] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:02:12] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/c1d81203f52c...024175046ab9 [02:02:15] [02miraheze/ManageWiki] 07AgentIsai 030241750 - Use Html [02:02:20] !log [www-data@test131] starting deploy of {'l10n': True, 'folders': 'w/extensions/ManageWiki'} to all [02:02:24] !log [www-data@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:02:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.44, 6.93, 6.82 [02:02:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.57, 3.98, 3.56 [02:04:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.46, 5.17, 3.84 [02:04:21] miraheze/ManageWiki - AgentIsai the build has errored. [02:04:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.26, 4.12, 3.66 [02:05:50] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:05:51] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/024175046ab9...6ffe092039d0 [02:05:54] [02miraheze/ManageWiki] 07AgentIsai 036ffe092 - Define $out [02:06:29] !log [www-data@test131] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [02:06:30] !log [www-data@test131] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [02:06:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.16, 3.82, 3.61 [02:06:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.57, 3.76, 3.56 [02:08:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 7.40, 7.05 [02:09:12] miraheze/ManageWiki - AgentIsai the build has errored. [02:10:04] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.89, 3.10, 3.34 [02:10:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 6.92, 6.91 [02:12:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.75, 7.48, 7.10 [02:13:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.86, 6.91, 6.95 [02:15:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.42, 6.78 [02:16:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 2.70, 3.22 [02:18:34] [02ManageWiki] 07AgentIsai synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:18:37] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/6ffe092039d0...f1f1c31bf025 [02:18:38] [02miraheze/ManageWiki] 07AgentIsai 03f1f1c31 - Use $this->msg [02:19:51] [02miraheze/ManageWiki] 07github-actions[bot] pushed 031 commit to 03managewikidefaultpermissions-fix [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/f1f1c31bf025...e01047db149a [02:19:54] [02miraheze/ManageWiki] 07github-actions 03e01047d - CI: lint code to MediaWiki standards [02:19:56] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#397: Allow Special:ManageWikiDefaultPermissions to actually allow you to reset group defaults on wikis - 13https://github.com/miraheze/ManageWiki/pull/397 [02:21:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 6.92, 6.91 [02:21:47] miraheze/ManageWiki - AgentIsai the build passed. [02:24:46] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.21, 5.54, 4.16 [02:25:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.07, 6.22, 6.64 [02:32:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.58, 7.01, 6.87 [02:34:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.31, 6.74, 6.78 [02:35:01] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.40440559387207 seconds [02:48:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 6.88, 6.72 [02:50:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.35, 3.33, 3.88 [02:50:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.66, 7.78, 7.04 [02:52:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.52, 3.29, 3.77 [02:52:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.42, 7.13, 6.89 [02:54:00] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.66, 3.00, 3.61 [02:54:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.39, 6.46, 6.67 [02:59:42] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.32, 3.50, 3.01 [02:59:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.46, 2.94, 3.39 [03:01:40] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 3.77, 3.16 [03:03:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.76, 4.19, 3.76 [03:04:17] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp32.miraheze.org - CNAME FLAT [03:05:35] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.40, 3.36, 3.22 [03:11:26] !log [agent@mwtask141] starting deploy of {'folders': 'geoip'} to all [03:11:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.12, 3.63, 3.40 [03:11:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:43] !log [agent@mwtask141] starting deploy of {'folders': 'geoip'} to all [03:12:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:02] !log [agent@mwtask141] finished deploy of {'folders': 'geoip'} to all - SUCCESS in 18s [03:13:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.39, 4.50, 3.71 [03:17:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.15, 3.57, 3.54 [03:17:42] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/StopForumSpam/maintenance/updateDenyList.php --wiki=metawiki (END - exit=0) [03:17:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:36] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 6.91, 6.66 [03:21:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.34, 4.41, 3.83 [03:23:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.21, 6.21, 6.43 [03:29:06] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.81, 3.71, 3.81 [03:29:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.48, 3.26, 3.89 [03:35:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.34, 3.38, 3.71 [03:36:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.54, 5.20, 4.21 [03:39:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.81, 3.59, 3.72 [03:42:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.60, 3.86 [03:43:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.73, 4.27, 3.83 [03:43:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 6.84, 6.39 [03:45:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.35, 3.62, 3.64 [03:45:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.66, 6.94, 6.50 [03:46:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.37, 3.91, 3.92 [03:47:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.64, 6.45 [03:48:44] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.84, 3.23, 3.68 [03:49:43] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:50:42] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.63, 3.47, 3.69 [03:51:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.29, 3.90, 3.69 [03:53:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.88, 3.46, 3.56 [03:55:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.50, 2.73, 3.27 [03:57:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.14, 6.77 [03:58:34] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.61, 3.78, 3.87 [04:01:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.36, 6.27, 6.54 [04:04:34] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.37, 2.54, 3.33 [04:07:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.58, 3.82, 3.24 [04:15:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.56, 3.58, 3.49 [04:17:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 2.98, 3.28 [04:31:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 6.96, 6.55 [04:35:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.96, 6.22, 6.36 [04:51:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.83, 8.11, 7.09 [04:53:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.54, 7.00 [04:59:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.44, 7.78, 7.21 [05:01:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.29, 7.09 [05:03:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:07:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.87, 3.85, 3.02 [05:07:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.40, 6.51, 6.77 [05:09:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.69, 3.39, 2.95 [05:11:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.90, 7.03, 6.87 [05:15:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.32, 6.40, 6.70 [05:19:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.58, 7.13 [05:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.49, 7.29, 7.08 [05:23:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.29, 6.24, 6.73 [05:30:23] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.30, 7.55, 3.69 [05:33:09] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:36:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.73, 3.32, 3.03 [05:51:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.19, 3.16, 2.40 [05:53:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.65, 3.11, 2.46 [05:55:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.37, 3.93, 2.86 [06:01:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.32, 3.89, 3.26 [06:05:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.94, 4.25, 3.49 [06:09:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.79, 3.19, 3.26 [06:26:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.19, 5.48, 3.99 [06:27:13] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.17, 5.01, 2.74 [06:29:13] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.54, 3.67, 2.53 [06:30:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.42, 3.52, 3.49 [06:32:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.91, 2.99, 3.30 [06:37:36] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864147 seconds old and 1753 bytes [06:43:58] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2231287658 secs [06:46:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.65, 3.72, 3.25 [07:00:51] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.46, 3.61, 3.74 [07:08:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.67, 2.92, 3.34 [07:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 6.29, 5.81 [07:15:36] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.70, 6.36, 6.07 [07:15:58] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02095770836 secs [07:37:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.42, 3.35, 2.55 [07:39:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 3.03, 2.55 [07:50:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 7.43, 6.62 [07:52:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.16, 6.68, 6.45 [08:02:18] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.85, 3.55, 3.00 [08:02:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.87, 6.02, 3.53 [08:04:14] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.87, 3.39, 3.01 [08:04:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.05, 5.10, 3.52 [08:10:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 8.00, 7.09 [08:18:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.29, 6.70 [09:50:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.47, 4.61, 2.40 [09:51:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 6.90, 6.38 [09:52:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.89, 3.44, 2.24 [09:55:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.66, 6.62, 6.37 [10:13:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 6.92, 6.47 [10:17:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.56, 6.45, 6.40 [10:35:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 7.07, 6.61 [10:37:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.16, 6.28, 6.37 [10:49:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 7.08, 6.66 [10:53:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.50, 6.52 [11:00:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.66, 7.09 [11:01:23] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:03:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.45, 6.52, 6.75 [11:07:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.62, 2.84, 1.90 [11:07:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.90, 6.71, 6.84 [11:09:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.17, 6.97 [11:11:03] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 2.25, 1.89 [11:11:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 6.85, 6.88 [11:13:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 6.96, 6.89 [11:15:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.28, 6.66 [11:31:05] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:49:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.19, 6.37 [11:51:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.77, 7.79, 6.71 [11:57:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.98, 6.50, 6.47 [12:00:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:18:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.20, 6.79 [12:22:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 8.07, 7.23 [12:24:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.84, 7.26 [12:26:12] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.124391675 secs [12:30:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.14, 7.02 [12:30:31] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:32:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.31, 6.64, 6.84 [12:34:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.20, 7.01 [12:36:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 7.00, 6.97 [12:41:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:41:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 7.13, 6.97 [12:43:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.09, 6.31, 6.68 [12:47:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.266 second response time [12:47:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 6.82, 6.82 [12:49:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.94, 7.78, 7.16 [12:51:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:51:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.51, 3.34, 2.36 [12:51:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.17, 7.01 [12:55:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 3.51, 2.64 [12:56:12] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08570191264 secs [12:57:04] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.61, 2.74, 2.46 [13:01:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.69, 6.36, 6.73 [13:11:46] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:51] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.497 second response time [13:23:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:26:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.06, 7.02, 6.55 [13:29:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 7.27, 6.79 [13:30:43] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:55] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:33:02] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [13:34:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.936 second response time [13:41:31] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:36] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.100 second response time [13:48:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:51:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.70, 8.58, 7.68 [13:53:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.89, 7.52 [14:02:17] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:02:32] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:03:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.02, 6.16, 6.80 [14:04:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.68, 7.83, 7.33 [14:09:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.08, 7.21, 7.17 [14:10:10] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.796 second response time [14:14:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.119 second response time [14:17:01] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:18:54] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.87, 7.99, 7.40 [14:21:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 7.48, 7.28 [14:23:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 7.88, 7.44 [14:23:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.031 second response time [14:25:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.72, 7.43 [14:27:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.86, 8.07, 7.61 [14:29:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 7.60, 7.51 [14:29:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.92, 9.33, 8.12 [14:31:40] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [14:31:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.148 second response time [14:35:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 7.73, 7.70 [14:36:56] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.15, 3.48, 2.62 [14:39:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.25, 2.64, 2.41 [14:41:06] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.728 second response time [14:43:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 8.14, 7.83 [14:45:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.50, 7.63 [14:47:38] PROBLEM - cp33 NTP time on cp33 is CRITICAL: NTP CRITICAL: Offset 0.9093212485 secs [14:49:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.17, 7.42 [14:51:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 7.33, 7.45 [14:51:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.0009304583073 secs [14:56:12] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.123534143 secs [15:01:02] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:07:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.05, 7.17, 7.19 [15:07:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:53] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.63, 2.27, 1.35 [15:09:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 6.57, 6.95 [15:09:27] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.356 second response time [15:10:51] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.95, 3.10, 2.30 [15:10:53] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.65, 2.93, 1.69 [15:11:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.31, 7.44, 7.21 [15:12:12] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09486716986 secs [15:12:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.92, 2.98, 2.35 [15:14:52] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.74, 3.14, 2.10 [15:17:01] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:17:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.95, 7.69 [15:19:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.42, 8.05, 7.77 [15:25:34] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.059 second response time [15:27:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.27, 7.83, 7.96 [15:30:12] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1552299559 secs [15:31:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.31, 7.90, 7.92 [15:33:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.70, 7.92, 7.93 [15:33:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.87, 3.66, 3.18 [15:33:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.52, 4.09, 3.42 [15:37:12] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.54, 3.46, 3.26 [15:39:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 2.83, 3.06 [15:43:26] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.504 second response time [15:46:12] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08687624335 secs [15:54:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.47, 3.58, 2.91 [15:56:12] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1536634862 secs [15:56:52] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/353e3f0940c2...f6886e100fba [15:56:54] [02miraheze/ssl] 07MirahezeSSLBot 03f6886e1 - Bot: Add SSL cert for wiki.walkscape.app [15:56:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 3.01, 2.78 [15:57:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.51, 7.20 [15:59:06] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.937 second response time [15:59:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.27, 7.08, 7.08 [16:05:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.56, 4.65, 3.51 [16:06:05] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:12] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09078097343 secs [16:07:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.94, 6.39, 6.73 [16:08:09] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.770 second response time [16:11:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 3.14, 3.27 [16:13:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:15:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.99, 4.20, 3.73 [16:17:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.37, 3.96, 3.70 [16:19:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.04, 4.19, 3.83 [16:21:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.25, 3.87, 3.76 [16:24:40] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.806 second response time [16:28:59] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.81, 3.94, 3.77 [16:30:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:31:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.357 second response time [16:31:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.17, 3.18, 3.51 [16:32:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:33:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.69, 2.78, 3.33 [16:35:19] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.27, 4.59, 4.05 [16:41:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.85, 3.49, 3.74 [16:42:07] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1116303802 secs [16:43:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.81, 4.51, 4.09 [16:43:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.72, 3.39, 2.89 [16:44:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.14, 6.54 [16:44:45] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.467 second response time [16:45:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.88, 3.55, 3.79 [16:45:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.80, 3.19, 2.88 [16:45:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.11, 6.81, 6.50 [16:49:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.03, 7.56, 6.85 [16:51:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.40, 3.98, 3.79 [16:51:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.09, 4.43, 3.43 [16:53:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.31, 3.99, 3.39 [16:53:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.87, 7.40, 6.99 [16:55:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 3.91, 3.83 [16:56:04] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.07456099987 secs [16:57:44] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 1 backends are down. mw121 [16:57:56] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.053 second response time [16:58:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.24, 2.89, 3.16 [16:59:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.00, 7.53, 7.12 [16:59:44] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [17:01:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.66, 7.19, 7.04 [17:02:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.972 second response time [17:02:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.64, 3.00, 3.37 [17:03:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.19, 6.28, 6.72 [17:04:17] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.33, 3.83, 3.62 [17:10:22] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.13, 2.89, 1.56 [17:10:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.97, 5.41, 2.81 [17:10:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.90, 3.13, 3.39 [17:12:22] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.79, 2.62, 1.63 [17:12:44] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.87, 4.61, 2.84 [17:15:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:15:30] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.595 second response time [17:15:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.65, 3.30 [17:16:13] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.69, 3.14, 2.05 [17:17:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.38, 3.19, 3.18 [17:18:12] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.37, 2.38, 1.90 [17:20:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.44, 8.23, 7.21 [17:21:36] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.79, 6.78, 4.65 [17:22:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.63, 7.10 [17:27:50] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.77, 3.91, 3.27 [17:29:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.45, 3.11, 3.05 [17:33:07] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.647 second response time [17:35:40] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:37:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.23, 7.87, 7.30 [17:39:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.92, 7.95, 7.41 [17:40:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.09, 3.17, 3.83 [17:43:22] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:07]