[00:02:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.44, 4.63, 3.67 [00:02:46] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.59, 3.40, 2.97 [00:03:07] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.99, 7.16, 4.04 [00:03:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.19, 7.53, 6.75 [00:03:49] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.10, 3.54, 1.85 [00:05:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.11, 6.67 [00:06:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.43, 3.02, 2.93 [00:07:23] [02CreateWiki] 07Universal-Omega closed pull request 03#404: Add new hook to directly pass private group name to - 13https://github.com/miraheze/CreateWiki/pull/404 [00:07:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.21, 10.21, 9.33 [00:07:51] miraheze/CreateWiki - Universal-Omega the build passed. [00:09:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.29, 3.47, 3.24 [00:09:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.40, 9.69, 9.25 [00:11:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.31, 6.51, 6.63 [00:11:07] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.36, 7.92, 5.80 [00:11:49] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.78, 3.89, 2.81 [00:12:33] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.89, 3.65, 3.70 [00:13:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.95, 6.11, 5.38 [00:13:49] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.69, 3.08, 2.64 [00:14:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.38, 4.00, 3.83 [00:15:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.97, 3.51, 3.36 [00:18:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 3.99, 3.90 [00:19:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.77, 3.28, 3.32 [00:20:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.23, 4.63, 4.14 [00:22:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.45, 3.94, 3.96 [00:22:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.77, 7.02, 6.66 [00:23:07] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.22, 7.93, 6.08 [00:24:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.66, 5.18, 4.42 [00:24:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.64, 8.66, 7.29 [00:25:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.55, 6.43, 5.77 [00:26:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.02, 7.33, 6.99 [00:26:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.98, 4.00, 3.21 [00:28:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.23, 3.18, 3.00 [00:32:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.32, 6.37, 6.71 [00:36:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.62, 3.90 [00:38:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.28, 10.72, 9.90 [00:39:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.43, 3.24, 3.17 [00:41:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.40, 3.32, 3.21 [00:43:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.91, 9.66, 9.76 [00:45:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.57, 6.39, 6.37 [00:46:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.02, 2.85, 3.38 [00:49:21] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.46, 3.77, 3.15 [00:49:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.92, 6.50, 6.45 [00:51:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.90, 2.97, 2.93 [01:10:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.76, 3.62, 3.08 [01:11:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.83, 3.38, 3.12 [01:12:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.33, 4.75, 3.52 [01:16:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.42, 6.97, 6.46 [01:16:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.80, 10.26, 9.44 [01:16:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.09, 3.63, 3.38 [01:17:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.98, 7.84, 6.83 [01:18:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.15, 9.92, 9.45 [01:19:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.95, 3.26, 3.21 [01:19:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.52, 6.83 [01:20:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.41, 3.41, 3.09 [01:22:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.82, 3.10, 3.01 [01:22:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.87, 3.32, 3.32 [01:25:07] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.92, 6.05, 3.51 [01:25:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.97, 6.05, 6.41 [01:26:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.94, 3.93, 3.56 [01:27:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.49, 4.19, 3.14 [01:28:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 3.15, 3.33 [01:36:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 6.32, 6.22 [01:38:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.56, 6.33 [01:52:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.64, 6.81 [01:56:50] [02ManageWiki] 07Universal-Omega opened pull request 03#401: ManageWikiSettings: support settings not in $wgManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/401 [01:57:20] [02ManageWiki] 07Universal-Omega synchronize pull request 03#401: ManageWikiSettings: support settings not in $wgManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/401 [01:57:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.02, 7.06, 6.93 [02:00:18] miraheze/ManageWiki - Universal-Omega the build passed. [02:01:41] miraheze/ManageWiki - Universal-Omega the build passed. [02:01:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.18, 8.32, 7.40 [02:03:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 7.32, 7.14 [02:04:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.01, 11.45, 9.68 [02:04:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.75, 5.39, 3.80 [02:05:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.22, 6.19, 6.74 [02:06:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 10.15, 9.43 [02:08:07] [02ManageWiki] 07AgentIsai closed pull request 03#401: ManageWikiSettings: support settings not in $wgManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/401 [02:08:10] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4c3cac1cb052...ed6341c9119d [02:08:13] [02miraheze/ManageWiki] 07Universal-Omega 03ed6341c - ManageWikiSettings: support settings not in $wgManageWikiSettings (#401) [02:08:14] [02WikiDiscover] 07AgentIsai closed pull request 03#98: Support excluding specific wikis from Special:WikiDiscover - 13https://github.com/miraheze/WikiDiscover/pull/98 [02:08:17] [02miraheze/WikiDiscover] 07AgentIsai pushed 033 commits to 03master [+0/-0/±4] 13https://github.com/miraheze/WikiDiscover/compare/36ef1ff22649...4679e6979625 [02:08:20] [02miraheze/WikiDiscover] 07Universal-Omega 0348704e3 - Support excluding specific wikis from Special:WikiDiscover [02:08:23] [02miraheze/WikiDiscover] 07Universal-Omega 0304a06f5 - Add config [02:08:24] [02miraheze/WikiDiscover] 07AgentIsai 034679e69 - Merge pull request #98 from Universal-Omega/patch-7 [02:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.70, 7.05, 6.97 [02:10:38] miraheze/WikiDiscover - AgentIsai the build has errored. [02:12:02] miraheze/ManageWiki - AgentIsai the build passed. [02:12:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.39, 3.94, 3.86 [02:15:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.53, 7.31, 7.07 [02:17:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.48, 3.23 [02:18:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.28, 4.40, 3.98 [02:20:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.50, 3.63, 3.75 [02:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 7.22, 7.20 [02:24:05] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.41, 2.62, 3.28 [02:29:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.32, 5.92, 6.68 [02:35:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.31, 3.88, 3.41 [02:37:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.10, 3.65, 3.50 [02:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 6.88, 6.91 [02:37:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 3.32, 3.26 [02:38:17] PROBLEM - nexus.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address nexus.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:42:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.60, 3.99, 3.00 [02:43:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.84, 11.08, 9.94 [02:44:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.12, 3.54, 2.95 [02:45:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.78, 3.57, 3.59 [02:45:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.00, 9.64, 9.55 [02:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.50, 7.27, 7.03 [02:45:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.81, 3.42, 3.31 [02:46:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.74, 2.83, 2.76 [02:47:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.31, 2.87, 3.11 [02:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.48, 6.81, 6.95 [02:51:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.68, 5.75, 6.55 [02:51:56] PROBLEM - nexus.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nexus.wiki could not be found [02:54:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.51, 3.96, 3.22 [02:55:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.51, 3.20, 3.15 [02:56:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.87, 3.76, 3.25 [02:57:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.16, 3.10, 3.11 [02:58:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.21, 4.44, 3.52 [02:59:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.53, 3.16, 3.36 [03:00:27] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 24 seconds old and 275 bytes [03:03:32] [02WikiDiscover] 07Universal-Omega opened pull request 03#99: Use ManageWikiSettings for WikiDiscoverExclude - 13https://github.com/miraheze/WikiDiscover/pull/99 [03:04:05] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#99: Use ManageWikiSettings for WikiDiscoverExclude - 13https://github.com/miraheze/WikiDiscover/pull/99 [03:05:05] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1096661687 secs [03:06:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.01, 3.23, 3.48 [03:07:22] miraheze/WikiDiscover - Universal-Omega the build passed. [03:07:58] miraheze/WikiDiscover - Universal-Omega the build passed. [03:08:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.41, 2.62, 3.22 [03:12:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.67, 10.76, 9.55 [03:13:02] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08219584823 secs [03:13:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.71, 4.04, 3.55 [03:14:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.93, 10.36, 9.55 [03:15:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.91, 3.99, 3.59 [03:15:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 3.50, 3.37 [03:16:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.88, 10.08, 9.57 [03:17:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 2.83, 3.14 [03:18:37] [02WikiDiscover] 07AgentIsai closed pull request 03#99: Use ManageWikiSettings for WikiDiscoverExclude - 13https://github.com/miraheze/WikiDiscover/pull/99 [03:18:38] [02miraheze/WikiDiscover] 07AgentIsai pushed 033 commits to 03master [+0/-0/±4] 13https://github.com/miraheze/WikiDiscover/compare/4679e6979625...8ef129339e90 [03:18:40] [02miraheze/WikiDiscover] 07Universal-Omega 03ef9d70b - Use ManageWikiSettings for WikiDiscoverExclude [03:18:43] [02miraheze/WikiDiscover] 07Universal-Omega 03585a59d - Remove [03:18:45] [02miraheze/WikiDiscover] 07AgentIsai 038ef1293 - Merge pull request #99 from Universal-Omega/patch-8 [03:20:55] miraheze/WikiDiscover - AgentIsai the build has errored. [03:21:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.15, 3.34 [03:24:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 6.93, 6.26 [03:26:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.31, 6.04, 6.02 [03:31:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.98, 3.60, 3.41 [03:33:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.00, 3.37, 3.35 [03:34:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.74, 6.54, 6.02 [03:36:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.53, 6.08 [03:37:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.10, 3.22, 2.61 [03:38:11] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.58, 5.77, 5.85 [03:39:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.39, 2.91, 2.57 [03:48:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.27, 3.74, 3.06 [03:50:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.61, 3.63, 3.11 [03:52:51] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.01, 3.20, 3.00 [03:58:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.33, 9.94, 9.00 [04:00:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.37, 6.48 [04:00:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.80, 4.17, 3.62 [04:02:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.90, 9.36, 9.03 [04:06:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.59, 3.70, 3.68 [04:06:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.16, 10.00, 8.58 [04:08:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.20, 6.79, 6.58 [04:08:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.82, 9.02, 8.38 [04:10:38] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.25, 3.10, 3.39 [04:11:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.35, 3.72, 3.31 [04:13:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.22, 3.54, 3.30 [04:14:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.89, 10.46, 9.85 [04:15:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.73, 3.28, 3.23 [04:16:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 6.81, 6.53 [04:18:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.11, 9.83, 9.76 [04:19:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.35, 6.56, 6.48 [04:23:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.54, 2.68 [04:24:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.41, 10.29, 9.90 [04:27:07] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.99, 5.71, 3.22 [04:27:53] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 3.01, 2.72 [04:29:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.85, 4.95, 3.28 [04:30:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.34, 12.51, 10.92 [04:32:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 10.64, 9.65 [04:34:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.48, 12.73, 10.50 [04:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.94, 7.06, 6.61 [04:39:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.82, 6.22, 6.35 [04:41:10] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.20, 6.59, 5.78 [04:41:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.05, 4.13, 3.54 [04:43:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.48, 6.61, 5.89 [04:43:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.42, 3.88, 3.53 [04:44:46] [02WikiDiscover] 07Universal-Omega opened pull request 03#100: Fix CI - 13https://github.com/miraheze/WikiDiscover/pull/100 [04:47:01] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.92, 7.02, 6.23 [04:47:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.37, 3.39, 3.39 [04:47:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.01, 11.80, 11.47 [04:48:21] miraheze/WikiDiscover - Universal-Omega the build passed. [04:48:59] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.45, 8.04, 6.68 [04:49:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.12, 12.16, 11.60 [04:50:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.52, 7.09, 6.62 [04:51:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 11.61, 11.50 [04:52:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.82, 6.45, 6.43 [04:54:53] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.76, 7.71, 7.04 [04:56:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 6.95, 6.66 [05:00:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.42, 6.53 [05:01:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.94, 10.73, 11.92 [05:04:42] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.30, 6.13, 6.63 [05:05:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.82, 11.56, 11.92 [05:06:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.01, 6.78 [05:07:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 10.29, 11.41 [05:09:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.38, 10.88, 10.61 [05:09:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.30, 11.50, 11.69 [05:10:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.54, 6.64 [05:11:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.35, 10.61, 10.52 [05:14:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 7.18, 6.92 [05:16:28] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.66, 7.46, 7.05 [05:16:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.36, 6.58, 6.73 [05:16:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.67, 9.75, 10.18 [05:18:28] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.61, 7.38, 7.07 [05:20:29] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.21, 7.95, 7.31 [05:20:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 6.95, 6.84 [05:20:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 10.08, 10.29 [05:22:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.25, 6.26, 6.59 [05:24:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.67, 9.17, 9.96 [05:27:40] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:29:46] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 6.145 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [05:31:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.52, 9.42, 11.51 [05:33:07] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.009 second response time [05:33:22] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.010 second response time [05:33:42] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.009 second response time [05:33:58] PROBLEM - db121 SSH on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:35:11] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:35:16] PROBLEM - db121 Puppet on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:35:16] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:35:17] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [05:35:31] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [05:35:40] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.010 second response time [05:35:46] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw141 [05:35:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.40, 5.53, 9.43 [05:36:06] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 7 backends are down. mw121 mw122 mw131 mw132 mw141 mw142 mediawiki [05:36:16] PROBLEM - db121 MariaDB Connections on db121 is UNKNOWN: [05:36:43] PROBLEM - db121 conntrack_table_size on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:37:33] PROBLEM - db121 NTP time on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:37:42] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.472 second response time [05:37:54] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 47% [05:38:39] PROBLEM - db121 Backups SQL on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:39:19] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.668 second response time [05:39:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.368 second response time [05:39:33] PROBLEM - db121 ferm_active on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:39:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.441 second response time [05:39:54] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 39% [05:40:15] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 1.076 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [05:40:15] PROBLEM - db121 MariaDB on db121 is CRITICAL: Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 104 [05:40:16] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 49 minutes ago with 0 failures [05:40:36] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 0 seconds old and 222134 bytes [05:41:11] PROBLEM - db121 MariaDB Connections on db121 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [05:41:21] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.554 second response time [05:41:30] RECOVERY - db121 ferm_active on db121 is OK: OK ferm input default policy is set [05:41:32] RECOVERY - db121 conntrack_table_size on db121 is OK: OK: nf_conntrack is 0 % full [05:41:55] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.541 second response time [05:42:04] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:42:29] RECOVERY - db121 NTP time on db121 is OK: NTP OK: Offset -0.001354902983 secs [05:43:32] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [05:44:03] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 221 Threads: 15 Questions: 24444 Slow queries: 1851 Opens: 2119 Open tables: 2113 Queries per second avg: 110.606 [05:44:06] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [05:44:43] !log [void@db121] set read_only=0 [05:44:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:05] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 3.4%Current connections: 17 [05:45:17] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [05:45:46] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [05:46:42] [02miraheze/puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-2 13https://github.com/miraheze/puppet/commit/3c0bfdf2cf6dc2ec0e01d92d88cd7b3fec53fa9e [05:46:44] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-2 - 13https://github.com/miraheze/puppet [05:48:37] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3c0bfdf2cf6d...1ba2f663c54b [05:48:40] [02miraheze/puppet] 07The-Voidwalker 031ba2f66 - mariadb remove read-only on startup [05:48:48] [02puppet] 07The-Voidwalker opened pull request 03#3215: mariadb remove read-only on startup - 13https://github.com/miraheze/puppet/pull/3215 [06:04:22] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/2e72dc6f023e...8cacfdff13d9 [06:04:25] [02miraheze/CreateWiki] 07AgentIsai 038cacfdf - Don't count account creations as activity [06:06:47] !log [agent@mwtask141] starting deploy of {'folders': 'w/extensions/CreateWiki'} to all [06:06:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:01] !log [agent@mwtask141] finished deploy of {'folders': 'w/extensions/CreateWiki'} to all - SUCCESS in 13s [06:07:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 6.58, 5.74 [06:09:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.57, 5.86, 5.59 [06:10:26] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 3.04, 1.90, 1.15 [06:13:55] miraheze/CreateWiki - AgentIsai the build passed. [06:14:26] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.26, 1.61, 1.20 [06:23:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.44, 3.18, 2.65 [06:23:26] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.48, 3.82, 2.55 [06:24:28] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 1.29, 1.69, 7.70 [06:25:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.32, 3.22, 2.73 [06:25:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.58, 4.00, 2.79 [06:25:33] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:27:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.23, 3.00, 2.57 [06:28:28] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.96, 1.27, 6.14 [06:30:47] [02puppet] 07AgentIsai opened pull request 03#3216: Increase SFS database list range - 13https://github.com/miraheze/puppet/pull/3216 [06:31:31] [02puppet] 07AgentIsai synchronize pull request 03#3216: Increase SFS database list range - 13https://github.com/miraheze/puppet/pull/3216 [06:34:41] [02puppet] 07AgentIsai synchronize pull request 03#3216: Increase SFS database list range - 13https://github.com/miraheze/puppet/pull/3216 [06:36:48] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/49aa8d598913...ac97155d08d8 [06:36:51] [02miraheze/mw-config] 07AgentIsai 03ac97155 - SFS: 30 days -> 90 days [06:37:58] miraheze/mw-config - AgentIsai the build passed. [06:41:15] !log [agent@mwtask141] starting deploy of {'folders': 'w/extensions/IPInfo'} to all [06:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:24] !log [agent@mwtask141] finished deploy of {'folders': 'w/extensions/IPInfo'} to all - SUCCESS in 9s [06:41:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:35] !log [@mwtask141] starting deploy of {'config': True} to all [06:41:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:43] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [06:41:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.01, 3.77, 3.22 [06:45:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.81, 3.43, 3.17 [06:47:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.68, 3.16, 3.09 [06:55:16] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:57:38] !log [@test131] starting deploy of {'config': True} to all [06:57:39] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:04:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.29, 6.35 [07:04:38] [02puppet] 07RhinosF1 commented on pull request 03#3215: mariadb remove read-only on startup - 13https://github.com/miraheze/puppet/pull/3215#issuecomment-1554116717 [07:06:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.07, 6.97, 6.35 [07:08:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.53, 6.59, 6.27 [07:10:01] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 10.63, 9.30 [07:11:41] [02puppet] 07Reception123 closed pull request 03#3216: Increase SFS database list range - 13https://github.com/miraheze/puppet/pull/3216 [07:11:42] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/3c0bfdf2cf6d...1cc9c5f06feb [07:11:44] [02miraheze/puppet] 07AgentIsai 031cc9c5f - Increase SFS database list range (#3216) [07:13:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.56, 3.30, 3.03 [07:15:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.86, 3.11, 2.99 [07:16:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.89, 11.12, 9.44 [07:22:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.54, 10.01, 9.53 [07:25:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.58, 11.36, 11.42 [07:28:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 6.86, 6.40 [07:29:53] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:30:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.31, 6.70, 6.39 [07:33:29] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:33:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.61, 8.82, 10.07 [07:35:54] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:37:18] @Reception123, @Agent: ^ [07:37:52] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:38:44] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:39:57] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:43:56] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:44:24] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [07:45:17] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_90_ipv46_all.txt] [08:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.01, 6.86, 6.46 [08:17:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.43, 6.45 [08:44:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.70, 10.50, 9.59 [08:46:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.69, 9.68, 9.38 [09:22:08] PROBLEM - mercy.theking1543.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mercy.theking1543.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:31:09] PROBLEM - o.kaoyu666.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for o.kaoyu666.cn could not be found [09:51:16] RECOVERY - mercy.theking1543.eu.org - reverse DNS on sslhost is OK: SSL OK - mercy.theking1543.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [09:53:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.32, 9.35, 8.14 [09:55:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.56, 8.85, 8.10 [10:11:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.69, 10.15, 8.69 [10:13:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.50, 9.41, 8.60 [10:51:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.74, 6.29 [10:55:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.87, 6.52, 6.33 [10:57:01] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:07:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.00, 6.54 [11:10:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.89, 12.36, 9.82 [11:13:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.79, 6.39, 6.54 [11:14:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.31, 10.78, 9.82 [11:16:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.74, 10.18, 9.72 [11:19:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.25, 10.55, 9.54 [11:22:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.64, 11.75, 10.44 [11:24:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.93, 11.50, 10.52 [11:25:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.08, 10.10, 9.68 [11:26:44] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:30:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.84, 9.98, 10.14 [11:33:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.14, 11.52, 10.42 [11:34:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 11.26, 10.61 [11:35:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 11.18, 10.42 [11:36:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.84, 11.42, 10.74 [11:38:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.70, 11.53, 10.85 [11:39:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.29, 9.85, 10.06 [11:44:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 6.97, 6.64 [11:45:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 11.08, 10.57 [11:48:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.08, 11.29, 11.42 [11:48:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.35, 6.70, 6.63 [11:49:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 11.26, 10.74 [11:50:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.79, 12.31, 11.75 [11:54:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.39, 7.92, 7.15 [11:56:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.03, 7.06, 6.92 [11:57:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.02, 11.84, 11.48 [11:58:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:00:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 11.42, 11.70 [12:07:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.42, 12.20, 11.90 [12:10:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.77, 6.27, 6.75 [12:15:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.30, 11.83, 11.99 [12:16:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.65, 10.92, 10.88 [12:17:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.16, 12.03, 12.05 [12:21:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 7.01, 6.75 [12:22:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.23, 11.87, 11.67 [12:23:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.76, 6.52, 6.58 [12:25:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.31, 11.71, 11.98 [12:27:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.70, 13.57, 12.60 [12:28:07] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:34:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.68, 12.94, 11.92 [12:40:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 11.58, 11.71 [12:47:42] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.15, 3.62, 3.20 [12:47:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.45, 10.46, 11.72 [12:49:37] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.12, 3.40, 3.17 [12:55:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.81, 11.44, 11.54 [12:57:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.08, 11.41, 11.53 [12:59:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.81, 12.36, 11.84 [13:00:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.24, 12.76, 11.46 [13:02:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.68, 2.96, 2.19 [13:04:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 3.30, 2.60 [13:06:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.81, 3.05, 2.58 [13:06:54] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.34, 2.79, 2.30 [13:07:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.79, 7.20, 6.54 [13:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.08, 8.34, 7.01 [13:13:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 7.68, 7.04 [13:17:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.13, 6.67, 6.79 [13:19:56] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.76, 3.44, 3.17 [13:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.57, 7.04, 6.93 [13:21:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.41, 3.10, 3.08 [13:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.31, 7.03 [13:29:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.38, 6.18, 6.64 [13:41:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 7.09, 6.73 [13:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 7.18, 6.89 [13:49:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.42, 6.72, 6.76 [13:51:28] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.99, 3.27, 2.71 [13:53:27] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.30, 3.52, 2.87 [13:55:26] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.40, 3.30, 2.86 [14:04:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.12, 6.88 [14:08:18] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.49, 6.67, 6.75 [14:25:07] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.18, 6.25, 4.06 [14:27:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.60, 5.21, 3.97 [14:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 8.82, 7.57 [14:31:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.13, 7.75, 7.44 [14:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.63, 7.20, 7.23 [14:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 7.27, 7.26 [14:38:30] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3dee30eb551f...1726c9a2c99f [14:38:31] [02miraheze/dns] 07Reception123 031726c9a - TESTAdd DNS zone for gil.wiki [14:38:54] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/1726c9a2c99f...9feafc97d969 [14:38:55] [02miraheze/dns] 07Reception123 039feafc9 - Revert "TESTAdd DNS zone for gil.wiki" [14:39:49] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.27, 2.96, 1.84 [14:41:49] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.31, 2.34, 1.76 [14:43:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.13, 5.97, 6.67 [14:51:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.31, 7.05 [14:54:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.47, 3.15, 2.57 [14:55:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.12, 7.00 [14:56:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.16, 2.84, 2.53 [14:57:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.49, 6.91, 6.94 [15:01:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.48, 8.08, 7.36 [15:03:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.36, 7.18 [15:07:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.27, 6.85, 6.95 [15:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 6.83, 6.91 [15:19:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.78, 8.22, 7.40 [15:21:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.74, 7.45 [15:25:19] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/95b22caaadab...5f339e58489c [15:25:20] [02miraheze/mediawiki] 07Reception123 035f339e5 - Update ManageWiki [15:26:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:26:49] !log [reception@mwtask141] starting deploy of {'world': True} to all [15:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.21, 8.48, 7.82 [15:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.81, 7.66 [15:33:09] !log [reception@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 379s [15:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 8.06, 7.80 [15:35:03] !log [reception@mwtask141] starting deploy of {'l10n': True, 'folders': 'w/extensions/ManageWiki'} to all [15:35:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:35:17] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.82, 3.32, 2.84 [15:37:17] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.26, 3.63, 3.01 [15:38:54] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:17] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.53, 3.23, 2.94 [15:40:50] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.551 second response time [15:45:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.60, 3.27, 2.91 [15:45:52] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.49, 2.54, 1.51 [15:47:51] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.43, 2.50, 1.63 [15:49:28] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 3.33, 3.09 [15:57:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.66, 3.28, 3.04 [16:01:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.57, 2.51, 2.78 [16:03:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 6.85, 7.81 [16:03:52] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:48] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.430 second response time [16:05:59] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.00, 7.15, 4.86 [16:06:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 8.33, 8.20 [16:11:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.46, 6.06, 5.28 [16:12:05] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.53, 3.12 [16:13:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.60, 7.95 [16:14:00] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.59, 4.09, 3.36 [16:15:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.72, 7.94 [16:15:55] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.90, 3.38 [16:19:45] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.18, 2.99, 3.13 [16:25:17] !log [reception@mwtask141] finished deploy of {'l10n': True, 'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 3013s [16:25:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:31:51] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.91, 3.45, 2.68 [16:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.09, 7.35, 7.86 [16:35:50] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.39, 4.24, 3.17 [16:42:50] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21725 MB (10% inode=99%); [16:43:30] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.92, 6.87, 5.94 [16:43:37] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.14, 7.81, 7.74 [16:44:05] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=purcalwiki --no-updates --username-prefix=w /home/reception/ImportDump/purcalwiki-20230518185406.xml (START) [16:44:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=purcalwiki --no-updates --username-prefix=w /home/reception/ImportDump/purcalwiki-20230518171828.xml (START) [16:44:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:50] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 28143 MB (14% inode=99%); [16:44:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:25] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.89, 6.75, 6.02 [16:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 7.57, 7.66 [16:45:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.63, 3.96, 3.07 [16:45:54] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=purcalwiki --no-updates --username-prefix=w /home/reception/ImportDump/purcalwiki-20230518171828.xml (END - exit=0) [16:45:56] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (START) [16:46:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=purcalwiki --no-updates --username-prefix=w /home/reception/ImportDump/purcalwiki-20230518185406.xml (END - exit=0) [16:46:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (START) [16:46:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:47:41] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.31, 3.73, 2.84 [16:47:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.084 second response time [16:47:50] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.70, 3.39, 2.95 [16:51:02] PROBLEM - os141 Disk Space on os141 is WARNING: DISK WARNING - free space: / 21288 MB (10% inode=99%); [16:51:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 20.77, 11.02, 5.77 [16:57:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:57:36] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.40, 3.83, 3.47 [16:59:35] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.21, 2.94, 3.19 [17:02:40] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:46] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.498 second response time [17:05:52] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:06:07] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (END - exit=0) [17:06:08] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (END - exit=0) [17:06:14] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=purcalwiki --active --update (END - exit=0) [17:06:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=purcalwiki --active --update (END - exit=0) [17:06:32] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:43] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:06:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:25] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:07:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.92, 6.07, 6.70 [17:07:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:09] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:15] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:10:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.207 second response time [17:11:30] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:12:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:12:50] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21786 MB (10% inode=99%); [17:13:13] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:14:59] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:15:41] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.41, 10.77, 8.16 [17:16:53] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.436 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [17:17:07] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:27] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:17:37] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 2.40, 7.66, 7.33 [17:21:18] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.505 second response time [17:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.42, 7.03 [17:21:49] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 1.47, 2.62, 3.96 [17:23:22] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.32, 4.51, 6.13 [17:25:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.44, 6.70 [17:25:49] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.48, 1.36, 3.13 [17:25:54] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.954 second response time [17:27:00] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.29, 12.90, 7.31 [17:27:51] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.92, 7.91, 7.25 [17:30:18] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.70, 7.24 [17:32:50] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 26485 MB (13% inode=99%); [17:33:01] RECOVERY - os141 Disk Space on os141 is OK: DISK OK - free space: / 26747 MB (13% inode=99%); [17:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.92, 7.60, 7.26 [17:36:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.266 second response time [17:37:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.76, 2.25, 3.92 [17:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.52, 6.65, 6.94 [17:39:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 7.13, 7.07 [17:40:59] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.15, 7.06 [17:47:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:49:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.77, 2.71, 3.35 [17:53:34] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.798 second response time [17:57:07] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.71, 5.56, 3.60 [17:57:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:58:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.79, 3.32, 3.95 [17:58:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.75, 4.99, 3.65 [18:01:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.03, 6.12, 6.80 [18:02:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.75, 3.37, 3.78 [18:04:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.81, 3.02, 3.60 [18:08:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.06, 2.67, 3.35 [18:09:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.66, 4.15, 3.56 [18:12:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.12, 5.18, 4.08 [18:12:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:13:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.29, 3.87, 3.60 [18:13:33] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.186 second response time [18:16:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.89, 3.78 [18:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.91, 7.04, 6.68 [18:17:58] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.52, 6.90 [18:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.35, 6.90 [18:23:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.86, 3.22, 3.38 [18:23:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.65, 6.68 [18:24:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 4.00, 3.46 [18:24:12] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.796 second response time [18:24:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.93, 2.80, 3.23 [18:27:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.61, 3.07, 3.23 [18:28:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.695 second response time [18:36:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:42] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:54] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.388 second response time [18:43:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 8.57, 7.45 [18:44:44] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.961 second response time [18:45:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.93, 7.34 [18:47:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.51, 8.54, 7.63 [18:47:36] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [18:49:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:55:29] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.579 second response time [18:56:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.04, 7.34, 7.56 [18:58:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.92, 8.97, 8.14 [19:00:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.70, 7.76, 7.79 [19:05:36] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.911 second response time [19:12:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.38, 8.47, 7.79 [19:14:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.66, 7.64, 7.56 [19:15:31] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.40, 3.04 [19:16:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:26] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.25, 3.14, 3.00 [19:17:29] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [19:18:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.33, 7.41 [19:22:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.74, 7.28, 7.45 [19:22:22] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.482 second response time [19:26:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.08, 7.38, 7.43 [19:28:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.63, 7.22, 7.35 [19:28:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.00, 3.77, 3.09 [19:29:28] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:26] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.15, 6.46, 3.85 [19:31:51] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.52, 2.91, 1.83 [19:32:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.08, 7.99, 7.57 [19:33:21] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.22, 6.46, 4.17 [19:33:49] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.74, 2.60, 1.86 [19:35:43] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.021 second response time [19:35:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.77, 7.43, 7.50 [19:36:00] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:06] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.926 second response time [19:40:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.42, 3.64, 3.70 [19:43:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.99, 7.63 [19:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 7.38, 7.46 [19:48:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.32, 2.70, 3.35 [19:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.20, 8.63, 7.90 [19:50:14] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.39, 7.59, 7.63 [19:55:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.69, 8.53, 7.97 [19:57:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.15, 7.74, 7.74 [19:58:37] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.254 second response time [20:04:19] [02mediawiki] 07wmfgerrit synchronize pull request 03#14709: Bump 1.40 to latest - 13https://github.com/miraheze/mediawiki/pull/14709 [20:05:06] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:05] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.007 second response time [20:09:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 7.00, 7.09 [20:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.20, 7.16 [20:13:30] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 16.79, 10.42, 8.29 [20:17:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.44, 4.42, 3.36 [20:19:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.10, 3.84, 3.26 [20:21:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.80, 4.00, 3.36 [20:23:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.53, 7.75, 7.87 [20:23:58] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.561 second response time [20:28:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:28:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.40, 3.80, 3.56 [20:29:26] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:30:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.323 second response time [20:30:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.11, 3.11, 3.33 [20:31:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.880 second response time [20:34:53] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.91, 3.85, 3.64 [20:35:56] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:36:52] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.35, 3.08, 3.39 [20:37:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.27, 7.14 [20:38:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.797 second response time [20:39:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.81, 7.45, 7.23 [20:41:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.47, 7.25 [20:42:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:31] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.717 second response time [20:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.69, 7.39, 7.30 [20:47:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.67, 7.72, 7.42 [20:51:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.32, 7.39, 7.43 [21:00:50] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21799 MB (10% inode=99%); [21:01:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.23, 5.91, 6.63 [21:03:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:05:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.76, 6.83 [21:06:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.03, 3.45, 2.85 [21:07:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.60, 6.03, 6.56 [21:08:36] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 2.95, 2.74 [21:08:50] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 22113 MB (11% inode=99%); [21:16:50] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21816 MB (10% inode=99%); [21:19:57] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:20:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.52, 6.76 [21:22:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.29, 6.72, 6.57 [21:25:01] PROBLEM - os141 Disk Space on os141 is WARNING: DISK WARNING - free space: / 20857 MB (10% inode=99%); [21:25:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.75, 3.88, 3.17 [21:27:01] RECOVERY - os141 Disk Space on os141 is OK: DISK OK - free space: / 22191 MB (11% inode=99%); [21:27:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.60, 3.34, 3.05 [21:31:01] PROBLEM - os141 Disk Space on os141 is WARNING: DISK WARNING - free space: / 21862 MB (10% inode=99%); [21:32:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.40, 7.80, 6.85 [21:36:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.03, 7.23, 6.88 [21:39:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.08, 3.51, 2.94 [21:40:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.96, 6.72, 6.75 [21:41:04] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.21, 2.88, 2.76 [21:43:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.20, 8.68, 7.59 [21:46:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.04, 3.69, 3.16 [21:47:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.69, 3.57, 3.21 [21:48:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 3.23, 3.06 [21:49:50] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:49:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.22, 3.34, 3.18 [22:01:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.77, 7.27, 7.77 [22:05:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.98, 4.87, 3.48 [22:09:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.24, 3.69, 3.33 [22:11:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.49, 2.99, 3.12 [22:17:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.43, 7.59, 7.37 [22:21:40] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [22:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 7.43, 7.37 [22:21:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.25, 3.53, 3.22 [22:23:56] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.59, 3.33, 3.19 [22:25:07] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.93, 5.17, 3.47 [22:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.93, 7.96, 7.54 [22:29:07] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.97, 6.39, 4.44 [22:29:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.59, 7.45 [22:32:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.09, 3.50, 3.30 [22:33:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.66, 8.89, 7.91 [22:34:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.02, 3.15, 3.21 [22:35:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.95, 7.64, 7.57 [22:39:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.60, 8.24, 7.79 [22:42:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.10, 3.85, 3.18 [22:43:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.62, 7.64, 7.65 [22:44:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.15, 3.29, 3.06 [22:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.50, 7.87, 7.72 [22:49:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.69, 7.68 [22:51:33] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:01:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.03, 7.83, 7.65 [23:05:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.32, 3.99, 3.16 [23:05:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.26, 7.16, 7.48 [23:07:31] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.25, 3.56, 3.10 [23:09:31] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 2.97, 2.94 [23:11:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.50, 8.43, 7.80 [23:13:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.57, 7.66, 7.60 [23:25:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 6.95, 7.13 [23:27:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.44, 6.26, 6.85 [23:28:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 10.33, 11.66 [23:29:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.20, 6.75 [23:34:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.87, 11.50, 11.56 [23:36:11] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=luxuryelevatorwiki --no-updates --username-prefix=wikia:luxury-elevator /home/agent/ImportDump/luxuryelevatorwiki-20230519174415.xml (START) [23:36:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.91, 10.86, 11.31 [23:37:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 10.17, 11.65 [23:38:19] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=luxuryelevatorwiki --no-updates --username-prefix=wikia:luxury-elevator /home/agent/ImportDump/luxuryelevatorwiki-20230519174415.xml (END - exit=0) [23:38:20] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=luxuryelevatorwiki (START) [23:38:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:38:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.99, 7.20, 6.83 [23:38:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.35, 11.57, 11.50 [23:40:21] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=luxuryelevatorwiki (END - exit=0) [23:40:23] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=luxuryelevatorwiki --active --update (END - exit=0) [23:40:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.73, 6.71 [23:40:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:40:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.76, 11.42, 11.44 [23:41:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.37, 12.80, 12.36 [23:43:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 11.59, 11.98 [23:44:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.27, 7.10, 6.99 [23:45:11] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.21, 3.76, 3.10 [23:47:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.06, 4.54, 3.48 [23:48:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.24, 11.22, 11.30 [23:49:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.52, 3.38, 3.18 [23:50:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.66, 11.13, 11.24 [23:51:18] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:53:53] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.39, 10.78, 11.11 [23:54:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.67, 11.39, 11.29 [23:57:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.98, 6.46, 6.76 [23:59:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.46, 11.24, 11.40