[00:01:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.95, 2.40, 1.77 [00:03:31] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.55, 5.08, 5.77 [00:03:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.70, 1.97, 1.61 [00:06:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.83, 3.43, 3.96 [00:07:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.99, 1.79 [00:07:29] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.96, 3.39, 4.92 [00:09:19] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.67, 1.69 [00:09:46] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.94, 1.62, 1.60 [00:10:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.00, 2.32, 3.40 [00:14:58] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 16.19, 6.87, 3.57 [00:18:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 11.73, 5.81, 4.25 [00:20:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.61, 4.00, 3.66 [00:21:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.92, 1.71 [00:21:20] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 19.20, 9.84, 6.65 [00:22:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.99, 3.01, 3.33 [00:23:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.93, 2.11, 1.82 [00:25:18] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.72, 5.89, 5.80 [00:25:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.93, 2.00, 1.81 [00:26:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.27, 1.73, 1.73 [00:28:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.56, 4.00 [00:28:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.55, 1.66 [00:29:21] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 11.28, 5.65, 5.53 [00:29:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.59, 1.46, 1.64 [00:30:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.84, 3.80, 4.03 [00:31:20] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.17, 4.49, 5.12 [00:32:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.70, 3.97 [00:33:19] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 4.20, 4.42, 5.01 [00:38:26] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.08, 3.18, 2.32 [00:38:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.44, 2.48, 3.39 [00:42:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.82, 1.81, 1.94 [00:46:51] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 10.16, 5.10, 2.97 [00:51:54] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 1.82, 1.88 [00:52:27] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.93, 2.07, 1.68 [00:53:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.14, 1.60, 1.79 [00:54:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.82, 1.64 [00:54:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.48, 3.34, 3.58 [00:55:45] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.85, 1.86 [00:56:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.84, 1.67 [00:56:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.37, 2.32, 3.17 [00:57:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.02, 1.58, 1.76 [00:59:13] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.74, 4.26, 2.21 [00:59:19] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.71, 1.60, 1.29 [00:59:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.93, 1.35, 1.65 [01:00:13] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.86, 1.72 [01:01:19] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.09, 1.38, 1.24 [01:02:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.20, 6.05, 4.43 [01:03:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.95, 2.81, 2.12 [01:03:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.78, 1.65, 1.71 [01:04:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.22, 1.67, 1.69 [01:05:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 1.80, 1.75 [01:07:24] PROBLEM - zenith.nsmbu.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zenith.nsmbu.net reverse DNS resolves to any-in-2001-4860-4802-32--15.1e100.net [01:07:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.42, 1.65, 1.70 [01:08:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.99, 1.87, 1.77 [01:08:29] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection. [01:09:58] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.84, 2.13, 1.87 [01:11:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.90, 1.81 [01:13:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.45, 1.85, 1.80 [01:15:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.58, 1.70 [01:15:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.31, 2.01, 1.87 [01:15:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.78, 3.48, 3.96 [01:16:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.33, 3.75, 2.53 [01:18:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.84, 2.94, 2.39 [01:19:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.70, 3.90 [01:21:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.63, 3.85 [01:21:50] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.99, 5.21, 4.10 [01:23:49] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.39, 4.59, 4.02 [01:25:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.81, 1.69, 1.94 [01:25:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.22, 3.60 [01:27:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 2.90, 3.45 [01:29:14] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.95, 4.44, 2.33 [01:29:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 2.61, 3.27 [01:31:18] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.67, 1.90, 1.87 [01:31:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.62, 2.12, 1.82 [01:33:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.89, 3.38, 2.49 [01:33:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 1.49, 1.72 [01:35:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.74, 1.74 [01:36:48] PROBLEM - zenith.nsmbu.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['eugene.ns.cloudflare.com.', 'kim.ns.cloudflare.com.'], 'CNAME': 'nsmbu.net.'} [01:39:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.99, 1.52, 1.66 [01:44:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.92, 1.37, 1.62 [01:47:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.79, 1.87, 1.71 [01:49:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.77, 1.69 [01:51:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.28, 3.01 [01:53:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.05, 1.85, 1.75 [01:54:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.72, 1.68 [01:55:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.90, 1.88, 1.78 [01:56:30] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.14, 5.78, 4.38 [01:56:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.51, 1.61 [01:58:29] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.29, 5.12, 4.31 [01:59:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.05, 1.59, 1.70 [02:00:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.49, 6.24, 4.85 [02:03:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.64, 1.68 [02:04:26] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.45, 4.71, 4.57 [02:05:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.67, 1.59, 1.65 [02:06:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.44, 1.80, 1.73 [02:07:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.78, 3.71, 3.77 [02:10:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.62, 1.68 [02:13:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.42, 2.62, 3.30 [02:22:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.83, 1.85, 1.64 [02:23:46] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.83, 1.76 [02:25:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.63, 1.70 [02:25:58] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.59, 1.60 [02:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.43, 3.29 [02:28:11] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.38, 5.00, 4.26 [02:29:13] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.80, 4.11, 1.97 [02:29:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 4.09, 3.55 [02:29:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.75, 1.68 [02:31:46] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.26, 1.66, 1.66 [02:32:09] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.00, 5.72, 4.76 [02:33:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.96, 3.99, 3.66 [02:33:13] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.12, 3.41, 2.28 [02:33:56] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [02:34:08] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.59, 4.82, 4.57 [02:35:13] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.62, 2.52, 2.09 [02:35:36] PROBLEM - zenith.nsmbu.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zenith.nsmbu.net reverse DNS resolves to any-in-2001-4860-4802-36--15.1e100.net [02:37:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 2.89, 3.27 [02:41:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 1.77, 1.66 [02:43:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.80, 1.68 [02:45:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.43, 1.56 [02:50:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 3.70, 3.22 [02:51:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.92, 2.30, 1.78 [02:52:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.52, 3.31, 3.14 [02:55:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.90, 2.78, 2.02 [02:56:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.71, 3.42 [02:57:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.84, 1.78 [02:58:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 3.17, 3.26 [03:01:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.97, 1.89 [03:02:53] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query housing.wiki. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:03:10] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK [03:03:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.57, 1.67 [03:05:37] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.58, 1.35, 1.66 [03:15:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 1.77, 1.69 [03:16:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.07, 4.26, 2.05 [03:19:37] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 2.16, 1.86 [03:20:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.08, 4.12, 3.24 [03:21:40] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.76, 5.63, 4.07 [03:22:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.75, 3.20 [03:23:50] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.12, 6.07, 4.42 [03:25:31] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.22, 1.75, 1.80 [03:25:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.77, 1.62 [03:25:45] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.15, 4.90, 4.19 [03:27:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.15, 3.51, 3.34 [03:27:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.07, 1.46, 1.68 [03:27:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.62, 1.59 [03:29:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 2.97, 3.16 [03:30:39] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:32:39] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:35:24] PROBLEM - zenith.nsmbu.net - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns-cloud-e2.googledomains.com.', 'ns-cloud-e3.googledomains.com.', 'ns-cloud-e1.googledomains.com.', 'ns-cloud-e4.googledomains.com.'], 'CNAME': 'nsmbu.net.'} [03:38:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.99, 4.27, 3.60 [03:38:19] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.96, 5.64, 4.47 [03:38:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.09, 1.75, 1.64 [03:40:15] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.25, 5.15, 4.42 [03:40:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.97, 1.64, 1.60 [03:42:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.65, 4.46, 4.27 [03:42:23] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.56, 1.72, 1.62 [03:44:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.34, 3.68, 3.66 [03:44:22] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.58, 1.59, 1.58 [03:46:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.24, 2.91, 3.38 [03:48:34] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.23, 1.03, 4.00 [03:50:03] PROBLEM - lab.wikimimo.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lab.wikimimo.com' expires in 15 day(s) (Sun 09 Oct 2022 03:42:01 GMT +0000). [03:51:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.67, 5.43, 4.18 [03:52:22] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.29, 0.61, 3.18 [03:52:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.82, 1.63 [03:54:44] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.54, 1.55 [03:55:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.56, 3.52, 3.69 [03:57:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.60, 3.68 [03:58:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f354175fa0d4...8a44c415db52 [03:58:19] [url] Comparing f354175fa0d4...8a44c415db52 · miraheze/ssl · GitHub | github.com [03:58:20] [02miraheze/ssl] 07MirahezeSSLBot 038a44c41 - Bot: Update SSL cert for lab.wikimimo.com [03:59:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.67, 3.70 [04:00:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 7.43, 4.85, 4.23 [04:01:04] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp30.miraheze.org - CNAME FLAT [04:01:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.47, 4.01 [04:04:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.94, 5.45, 4.57 [04:04:46] PROBLEM - zenith.nsmbu.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - zenith.nsmbu.net reverse DNS resolves to any-in-2001-4860-4802-38--15.1e100.net [04:06:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.36, 4.33, 4.27 [04:07:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.24, 3.68 [04:13:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.85, 2.65, 3.29 [04:19:44] RECOVERY - lab.wikimimo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'lab.wikimimo.com' will expire on Thu 22 Dec 2022 02:58:12 GMT +0000. [04:23:02] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.63, 1.55 [04:24:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.43, 4.19, 3.71 [04:25:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.42, 3.18 [04:26:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.90, 4.20, 3.79 [04:27:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.87, 1.41, 1.49 [04:27:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.01, 2.99, 3.06 [04:48:50] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.28, 1.67, 1.45 [04:49:35] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [04:49:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:50:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.54, 1.60, 1.45 [05:00:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.79, 2.87 [05:00:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.80, 4.32, 2.23 [05:02:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.28, 3.21, 2.77 [05:02:52] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.23, 3.04, 2.01 [05:06:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.65, 3.03 [05:07:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.10, 2.90 [05:33:50] PROBLEM - en.clockupwiki.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406757831573486 seconds [05:58:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.29, 2.47, 1.44 [06:03:16] PROBLEM - en.clockupwiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.clockupwiki.org All nameservers failed to answer the query. [06:03:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 4.30, 2.89 [06:04:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.12, 6.27, 3.79 [06:07:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.90, 3.09 [06:08:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.58, 3.48, 3.29 [06:09:31] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.50, 1.91, 1.42 [06:09:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 3.10, 2.90 [06:10:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.69, 4.51, 4.01 [06:10:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.26, 2.42, 2.92 [06:21:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.29, 2.86 [06:23:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 2.95, 2.77 [06:28:48] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 11.76, 7.91, 4.47 [06:45:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.90, 2.87, 1.75 [06:47:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.29, 2.28, 1.67 [06:48:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.68, 1.39, 3.65 [06:50:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.54, 1.10, 3.27 [07:00:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.63, 5.32, 3.85 [07:04:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.98, 3.76, 3.65 [07:06:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.30, 2.62, 3.24 [07:11:31] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.14, 1.64, 1.97 [07:17:31] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.12, 1.21, 1.65 [07:22:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 2.11, 1.33 [07:24:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.15, 1.94, 1.37 [07:26:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.48, 1.27 [07:45:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.82, 3.29, 1.82 [07:53:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.06, 3.02, 2.58 [07:57:31] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.23, 1.68, 1.34 [07:59:31] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.32, 1.51, 1.32 [08:30:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.90, 3.91, 2.37 [08:32:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.96, 2.79, 2.14 [09:30:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.04, 5.47, 2.99 [09:36:36] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:37:18] PROBLEM - cp23 SSH on cp23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:27] PROBLEM - cp23 PowerDNS Recursor on cp23 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:39:35] RECOVERY - cp23 PowerDNS Recursor on cp23 is OK: DNS OK: 6.548 seconds response time. miraheze.org returns 217.174.247.33,2a00:da00:1800:326::1 [09:41:27] RECOVERY - cp23 SSH on cp23 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [09:41:35] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [09:42:09] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:52] PROBLEM - mwtask141 php-fpm on mwtask141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:43:20] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 21.81, 12.01, 5.26 [09:43:23] PROBLEM - mwtask141 PowerDNS Recursor on mwtask141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:44:40] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:44:40] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [09:44:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:44:48] RECOVERY - mwtask141 php-fpm on mwtask141 is OK: PROCS OK: 3 processes with command name 'php-fpm7.4' [09:45:22] RECOVERY - mwtask141 PowerDNS Recursor on mwtask141 is OK: DNS OK: 0.103 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [09:45:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.97, 2.77, 1.88 [09:46:17] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 30911 bytes in 0.328 second response time [09:46:40] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [09:47:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.81, 2.37, 1.85 [09:53:33] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [09:53:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:57:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.60, 2.68, 3.95 [10:01:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 3.93, 3.01 [10:03:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.29, 2.88 [10:05:03] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.57, 2.63, 3.37 [10:06:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.38, 1.63, 3.80 [10:07:48] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.97, 5.46, 4.50 [10:10:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.92, 1.45, 3.25 [10:11:39] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.31, 3.97, 4.17 [10:16:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 3.54, 3.14 [10:18:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.12, 3.02 [10:22:34] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:27:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.90, 3.41 [10:28:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.33, 7.04, 5.31 [10:30:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.09, 5.69, 5.03 [10:31:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.35, 3.18, 3.29 [10:32:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.10, 4.84, 4.80 [10:35:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.47, 4.85, 3.89 [10:36:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.43, 6.61, 5.59 [10:37:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.96, 3.67 [10:38:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.96, 5.27, 5.22 [10:39:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.33, 3.11, 3.40 [10:40:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.43, 4.23, 4.83 [10:52:34] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:00:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.47, 3.27, 2.11 [11:02:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.93, 2.34, 1.91 [11:28:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.75, 2.86, 1.80 [11:30:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.76, 2.54, 1.83 [11:35:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.47, 3.03 [11:37:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.18, 2.97 [11:43:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.62, 3.19 [11:45:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.38, 4.35, 3.50 [11:48:05] RECOVERY - zenith.nsmbu.net - LetsEncrypt on sslhost is OK: OK - Certificate 'zenith.nsmbu.net' will expire on Mon 10 Oct 2022 05:53:48 GMT +0000. [11:53:52] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.62, 6.43, 4.90 [11:54:25] RECOVERY - zenith.nsmbu.net - reverse DNS on sslhost is OK: SSL OK - zenith.nsmbu.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [11:55:48] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.25, 5.40, 4.70 [11:57:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.17, 3.72, 3.90 [11:57:44] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.55, 4.39, 4.41 [12:03:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 3.87, 3.79 [12:07:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.75, 3.17, 3.55 [12:09:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.26, 3.52 [12:11:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.23, 2.53, 3.22 [12:15:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.33, 3.47, 1.99 [12:17:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.66, 3.53, 2.20 [12:19:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 9.33, 5.93, 3.26 [12:24:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.93, 4.27, 3.61 [12:25:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.83, 3.04, 2.88 [12:26:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.74, 3.46 [12:26:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.63, 3.64, 3.47 [12:27:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.37, 3.35 [12:31:20] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.89, 3.45, 2.32 [12:33:17] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.16, 2.59, 2.14 [12:42:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.14, 3.06 [12:44:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.09, 3.06 [12:48:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.57, 3.28 [12:50:28] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.09, 3.14 [12:58:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.13, 2.87, 2.30 [13:00:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.05, 2.52, 2.24 [13:17:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 12.05, 9.35, 5.48 [13:18:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.93, 6.45, 4.19 [13:20:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 3.27, 5.08, 3.95 [13:30:35] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [13:30:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:31:22] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.62, 3.43, 2.40 [13:33:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.22, 2.69, 2.25 [13:36:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.19, 7.02, 4.85 [13:42:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.97, 6.00, 5.42 [13:48:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.75, 5.85, 5.46 [13:50:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.34, 5.86, 5.53 [13:52:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.55, 6.38, 5.80 [13:53:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [13:53:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.30, 5.67, 5.58 [13:57:22] PROBLEM - source.erag.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'source.erag.eu.org' expires in 15 day(s) (Sun 09 Oct 2022 13:40:55 GMT +0000). [13:58:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8a44c415db52...6a76c22905ca [13:58:08] [url] Comparing 8a44c415db52...6a76c22905ca · miraheze/ssl · GitHub | github.com [13:58:09] [02miraheze/ssl] 07MirahezeSSLBot 036a76c22 - Bot: Update SSL cert for source.erag.eu.org [13:58:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.10, 5.34, 5.36 [14:08:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.30, 4.93, 5.64 [14:10:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.24, 6.27, 6.05 [14:14:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.04, 4.02, 2.67 [14:16:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.55, 3.61, 2.67 [14:18:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.97, 2.64, 2.43 [14:20:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.00, 5.00, 5.63 [14:24:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.68, 3.63, 4.92 [14:26:53] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Thu 22 Dec 2022 12:58:01 GMT +0000. [14:28:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.18, 2.83, 2.38 [14:30:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.63, 2.45, 2.31 [14:45:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.16, 3.58, 2.18 [14:47:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.09, 2.92, 2.10 [14:51:36] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 15.09, 9.49, 6.57 [14:52:53] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.43, 2.88, 2.00 [14:55:27] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.35, 5.86, 5.77 [14:56:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.74, 1.72 [14:57:22] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.59, 6.60, 6.07 [14:58:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.38, 1.52, 1.64 [14:59:18] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 4.89, 5.82, 5.83 [15:01:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.10, 6.26, 5.96 [15:14:42] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.55, 5.00, 5.75 [15:18:34] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 8.11, 5.35, 5.62 [15:20:29] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 5.82, 5.68, 5.73 [15:22:25] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 11.53, 8.18, 6.65 [15:28:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.81, 5.14, 5.81 [15:30:14] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 9.11, 6.86, 6.36 [15:41:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 9.32, 8.07 [15:45:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.53, 9.31, 8.40 [15:54:14] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 1.74, 3.18, 5.73 [15:58:14] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 0.74, 1.99, 4.66 [16:01:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 2.41, 3.90 [16:11:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.23, 9.50, 8.75 [16:13:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.53, 2.32, 3.19 [16:13:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 9.37, 8.82 [16:15:17] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 2.09, 1.72 [16:17:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.28, 1.70, 1.62 [16:20:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.85, 4.27, 2.61 [16:21:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.46, 1.90, 1.50 [16:23:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.96, 1.58 [16:24:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.45, 3.34 [16:24:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.63, 3.37, 2.70 [16:25:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.13, 1.71, 1.68 [16:26:07] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.91, 3.16 [16:27:11] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.22, 1.85, 1.73 [16:29:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.61, 1.56 [16:30:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 4.35, 3.69 [16:31:09] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.08, 1.62, 1.69 [16:31:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.84, 3.59 [16:35:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.81, 1.74, 1.71 [16:37:06] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.40, 1.59 [16:37:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.87, 2.90, 3.26 [16:55:56] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.92, 1.73 [16:57:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.68, 1.67 [17:03:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [17:03:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:08:50] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.50, 1.38 [17:10:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.84, 1.34, 1.34 [17:23:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.73, 1.52 [17:25:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.47, 1.45 [17:40:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.83, 4.25, 3.47 [17:42:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.81, 3.38 [17:46:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.56, 1.36 [17:47:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.85, 5.03, 3.91 [17:50:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.35, 1.54, 1.40 [17:53:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.20, 3.92, 3.91 [17:54:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.84, 1.60 [17:55:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.77, 3.84 [17:58:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.57, 1.55 [17:59:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.88, 3.87 [18:01:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 4.58, 4.14 [18:09:31] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 2.04, 1.60, 1.27 [18:12:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 2.88, 2.37 [18:14:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 2.68, 2.36 [18:25:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.32, 1.76, 1.59 [18:27:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 1.84, 1.62 [18:30:31] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [18:30:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.09, 3.35, 2.21 [18:31:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.88, 1.68 [18:32:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.65, 2.91, 2.18 [18:33:59] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.53, 1.97, 1.73 [18:35:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.89, 1.73 [18:40:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cebb8e6b7a40 [18:40:11] [02miraheze/puppet] 07Universal-Omega 03cebb8e6 - swift: allow Icinga2 access to firewall [18:40:12] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [18:40:13] [url] Page not found · GitHub · GitHub | github.com [18:40:14] [02puppet] 07Universal-Omega opened pull request 03#2858: swift: allow Icinga2 access to firewall - 13https://github.com/miraheze/puppet/pull/2858 [18:40:15] [url] Page not found · GitHub · GitHub | github.com [18:41:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.47, 1.62 [18:54:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.61, 2.35, 1.81 [18:56:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.10, 1.74, 1.65 [18:58:00] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.37, 1.52 [18:59:02] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK [19:11:31] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.02, 1.60, 1.92 [19:15:31] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.02, 1.25, 1.70 [19:23:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.80, 1.54 [19:24:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.86, 1.63 [19:25:00] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.32, 1.63, 1.51 [19:26:14] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.47, 1.51 [19:27:41] [02puppet] 07paladox reviewed pull request 03#2858 commit - 13https://github.com/miraheze/puppet/pull/2858#discussion_r979004906 [19:27:42] [url] swift: allow Icinga2 access to firewall by Universal-Omega · Pull Request #2858 · miraheze/puppet · GitHub | github.com [19:27:56] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.14, 3.34, 2.01 [19:29:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.32, 2.56, 1.89 [19:30:19] [02puppet] 07Universal-Omega synchronize pull request 03#2858: swift: allow Icinga2 access to firewall - 13https://github.com/miraheze/puppet/pull/2858 [19:30:20] [url] Page not found · GitHub · GitHub | github.com [19:30:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cebb8e6b7a40...e1cc2ef09718 [19:30:22] [url] Comparing cebb8e6b7a40...e1cc2ef09718 · miraheze/puppet · GitHub | github.com [19:30:22] [02miraheze/puppet] 07Universal-Omega 03e1cc2ef - Update swift.pp [19:30:30] [02puppet] 07Universal-Omega reviewed pull request 03#2858 commit - 13https://github.com/miraheze/puppet/pull/2858#discussion_r979006681 [19:30:31] [url] swift: allow Icinga2 access to firewall by Universal-Omega · Pull Request #2858 · miraheze/puppet · GitHub | github.com [19:35:59] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=monstergirlencyclopediawiki --r=Requested at [[phab:T9754 [19:36:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:43:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.41, 1.76, 1.52 [19:45:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.95, 1.69, 1.51 [19:47:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.56, 1.70, 1.54 [19:48:36] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.24, 3.46, 2.01 [19:53:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.86, 1.66 [19:54:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fbabdc9ac5ef [19:54:13] [02miraheze/puppet] 07Universal-Omega 03fbabdc9 - jobQueueCollector: add z-claimed and z-delayed jobs [19:54:14] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/puppet [19:54:15] [url] Page not found · GitHub · GitHub | github.com [19:54:16] [02puppet] 07Universal-Omega opened pull request 03#2859: jobQueueCollector: add z-claimed and z-delayed jobs - 13https://github.com/miraheze/puppet/pull/2859 [19:54:17] [url] Page not found · GitHub · GitHub | github.com [19:56:12] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.29, 3.33, 2.90 [19:59:06] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.55, 3.38, 2.22 [20:03:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.52, 1.70, 1.69 [20:07:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.72, 1.70 [20:08:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.85, 3.46, 3.56 [20:09:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.04, 2.02, 1.80 [20:10:52] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.55, 2.48, 3.18 [20:13:51] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.52, 1.77, 1.49 [20:15:50] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.26, 1.64, 1.47 [20:17:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.67, 1.99, 1.97 [20:19:47] [02puppet] 07paladox closed pull request 03#2858: swift: allow Icinga2 access to firewall - 13https://github.com/miraheze/puppet/pull/2858 [20:19:48] [url] Page not found · GitHub · GitHub | github.com [20:19:49] [02miraheze/puppet] 07paladox deleted branch 03Universal-Omega-patch-1 [20:19:50] [02puppet] 07paladox deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [20:19:51] ... [20:19:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1831f19fcc19...08d2f8f47457 [20:19:53] [url] Comparing 1831f19fcc19...08d2f8f47457 · miraheze/puppet · GitHub | github.com [20:19:53] [02miraheze/puppet] 07Universal-Omega 0308d2f8f - swift: allow Icinga2 access to firewall (#2858) [20:21:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.08, 2.31, 2.08 [20:21:47] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.20, 2.08, 1.71 [20:23:47] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.90, 1.69 [20:24:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.10, 2.63 [20:25:46] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.49, 2.50, 1.91 [20:26:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 2.80, 2.58 [20:30:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.61, 2.99 [20:30:31] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:31:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.95, 1.66, 1.74 [20:32:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 3.20, 2.92 [20:33:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.17, 1.68, 1.96 [20:33:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.37, 1.54, 1.68 [20:35:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.96, 2.03 [20:37:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.67, 1.45, 1.83 [20:38:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.60, 3.15 [20:41:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.11, 1.82, 1.72 [20:41:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.79, 3.30, 3.15 [20:43:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.13, 1.58 [20:43:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.25, 3.98 [20:44:07] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is WARNING: HTTP WARNING: HTTP/1.1 401 Unauthorized - 476 bytes in 0.063 second response time [20:47:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.64, 3.41, 2.30 [20:49:48] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.15, 3.74, 2.58 [20:51:48] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.85, 2.64, 2.32 [20:55:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 2.79, 3.36 [20:58:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 2.23, 1.90 [20:59:01] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - roblox-wiki.tk All nameservers failed to answer the query. [21:01:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/52291bfc2cef [21:01:32] [02miraheze/puppet] 07paladox 0352291bf - swift: mask service swift-*-replicator [21:01:33] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [21:01:34] [url] Page not found · GitHub · GitHub | github.com [21:01:35] [02puppet] 07paladox opened pull request 03#2860: swift: mask service swift-*-replicator - 13https://github.com/miraheze/puppet/pull/2860 [21:01:36] [url] Page not found · GitHub · GitHub | github.com [21:01:59] [02puppet] 07paladox edited pull request 03#2860: swift: mask service swift-*-replicator - 13https://github.com/miraheze/puppet/pull/2860 [21:02:00] ... [21:02:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.96, 1.85 [21:02:58] [02puppet] 07paladox synchronize pull request 03#2860: swift: mask service swift-*-replicator - 13https://github.com/miraheze/puppet/pull/2860 [21:02:59] ... [21:03:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/52291bfc2cef...481eb917f02d [21:03:00] [url] Comparing 52291bfc2cef...481eb917f02d · miraheze/puppet · GitHub | github.com [21:03:01] [02miraheze/puppet] 07paladox 03481eb91 - Update storage.pp [21:03:29] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.51, 1.68, 1.97 [21:04:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/481eb917f02d...055808474ab6 [21:04:29] [url] Comparing 481eb917f02d...055808474ab6 · miraheze/puppet · GitHub | github.com [21:04:29] [02miraheze/puppet] 07paladox 030558084 - Update ac.pp [21:04:31] [02puppet] 07paladox synchronize pull request 03#2860: swift: mask service swift-*-replicator - 13https://github.com/miraheze/puppet/pull/2860 [21:04:32] [url] Page not found · GitHub · GitHub | github.com [21:05:01] [02puppet] 07paladox closed pull request 03#2860: swift: mask service swift-*-replicator - 13https://github.com/miraheze/puppet/pull/2860 [21:05:02] [url] Page not found · GitHub · GitHub | github.com [21:05:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/08d2f8f47457...a0944c98845c [21:05:04] [url] Comparing 08d2f8f47457...a0944c98845c · miraheze/puppet · GitHub | github.com [21:05:04] [02miraheze/puppet] 07paladox 03a0944c9 - swift: mask service swift-*-replicator (#2860) [21:05:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:05:07] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [21:05:08] [url] Page not found · GitHub · GitHub | github.com [21:07:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.04, 1.64, 1.86 [21:08:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.49, 1.68 [21:09:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.35, 1.58, 1.81 [21:10:01] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:15:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.83, 2.51, 2.06 [21:15:48] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.13, 3.02, 1.93 [21:15:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.87, 3.40 [21:17:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.54, 3.33 [21:19:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 3.20, 3.23 [21:21:27] PROBLEM - cp22 PowerDNS Recursor on cp22 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:21:49] PROBLEM - cp22 SSH on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.72, 3.46 [21:24:11] PROBLEM - cp22 Puppet on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:24:42] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:25:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.10, 1.67, 1.92 [21:25:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.56, 3.39, 3.38 [21:25:56] RECOVERY - cp22 SSH on cp22 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:26:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.26, 2.91, 2.12 [21:26:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.66, 3.53, 2.40 [21:27:11] PROBLEM - cp22 conntrack_table_size on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:28:31] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [21:28:42] PROBLEM - db131 NTP time on db131 is WARNING: NTP WARNING: Offset 0.1223422885 secs [21:28:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.28, 3.20, 2.43 [21:30:15] PROBLEM - cp22 SSH on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:31:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.35, 1.70 [21:32:20] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [21:33:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [21:34:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.99, 1.99 [21:35:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:36:39] PROBLEM - cp22 ferm_active on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:37:32] PROBLEM - cp22 Stunnel HTTP for puppet141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:39:14] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.58, 1.64 [21:39:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [21:39:50] RECOVERY - cp22 Stunnel HTTP for puppet141 on cp22 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 8.484 second response time [21:41:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.78, 2.00, 1.78 [21:41:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:42:23] RECOVERY - cp22 ferm_active on cp22 is OK: OK ferm input default policy is set [21:42:25] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [21:42:28] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3361 bytes in 0.275 second response time [21:42:34] RECOVERY - cp22 SSH on cp22 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [21:42:51] PROBLEM - bast121 NTP time on bast121 is WARNING: NTP WARNING: Offset 0.1187093556 secs [21:43:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:44:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.47, 1.70 [21:44:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 9.24, 5.19, 3.87 [21:45:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.70, 1.94, 1.82 [21:46:53] PROBLEM - cp22 SSH on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.73, 1.77 [21:49:05] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:49:09] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.25, 1.87, 1.80 [21:49:26] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:49:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:da00:1800:326::1/cpweb [21:50:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.45, 1.66 [21:50:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 217.174.247.33/cpweb [21:50:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.43, 3.10, 3.37 [21:51:09] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.83, 1.80 [21:52:04] PROBLEM - cp22 ferm_active on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:55:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:57:06] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.30, 1.45, 1.64 [21:58:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.93, 1.75 [21:58:54] PROBLEM - test131 NTP time on test131 is WARNING: NTP WARNING: Offset 0.1144105494 secs [21:59:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [22:00:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.68, 1.68 [22:00:07] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:09] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3361 bytes in 5.005 second response time [22:04:54] PROBLEM - cp22 Stunnel HTTP for mon141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:26] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:36] PROBLEM - cp22 Stunnel HTTP for mw132 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:36] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:09:39] RECOVERY - cp22 Stunnel HTTP for mon141 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 36344 bytes in 7.801 second response time [22:09:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:10:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.37, 2.32, 1.90 [22:10:45] RECOVERY - cp22 Stunnel HTTP for mw132 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 3.590 second response time [22:11:08] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset 0.09834980965 secs [22:12:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.93, 1.80 [22:13:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [22:13:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [22:15:17] PROBLEM - cp22 Stunnel HTTP for reports121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:29] PROBLEM - cp22 Stunnel HTTP for mw142 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:09] PROBLEM - cp22 Stunnel HTTP for mail121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:42] PROBLEM - cp22 Stunnel HTTP for mwtask141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:43] PROBLEM - cp22 Stunnel HTTP for phab121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.89, 1.79 [22:18:42] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3361 bytes in 7.862 second response time [22:18:45] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.32, 3.79, 2.53 [22:18:51] RECOVERY - bast121 NTP time on bast121 is OK: NTP OK: Offset 0.09987017512 secs [22:19:25] PROBLEM - cp22 Stunnel HTTP for matomo131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:19:42] PROBLEM - cp22 Stunnel HTTP for mw131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.87, 1.80 [22:20:07] RECOVERY - cp22 Stunnel HTTP for reports121 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 9.632 second response time [22:20:22] RECOVERY - cp22 Stunnel HTTP for mw142 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 9.743 second response time [22:21:31] RECOVERY - cp22 Stunnel HTTP for phab121 on cp22 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 9.716 second response time [22:22:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 2.12, 1.91 [22:22:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 4.05, 3.52 [22:22:34] PROBLEM - cp22 Stunnel HTTP for puppet141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.86, 3.65, 2.85 [22:22:53] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.79, 2.02, 1.76 [22:23:00] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:09] PROBLEM - cp22 Stunnel HTTP for mw122 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:41] RECOVERY - cp22 Stunnel HTTP for mail121 on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 7.440 second response time [22:24:22] RECOVERY - cp22 Stunnel HTTP for mw131 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.980 second response time [22:24:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.00, 2.74, 2.62 [22:24:52] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.04, 1.65, 1.66 [22:25:00] PROBLEM - cp22 Stunnel HTTP for test131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:00] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3361 bytes in 7.308 second response time [22:25:05] RECOVERY - cp22 Stunnel HTTP for puppet141 on cp22 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 8.971 second response time [22:25:24] PROBLEM - cp22 Stunnel HTTP for mw132 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:53] PROBLEM - cp22 Stunnel HTTP for mw142 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:07] PROBLEM - cp22 Stunnel HTTP for mw121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:07] PROBLEM - cp22 Stunnel HTTP for mw141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:26:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.60, 3.47 [22:27:01] RECOVERY - cp22 Stunnel HTTP for matomo131 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 100159 bytes in 5.729 second response time [22:28:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.97, 1.96 [22:28:09] RECOVERY - db131 NTP time on db131 is OK: NTP OK: Offset 0.08946782351 secs [22:28:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.20, 3.33 [22:28:32] RECOVERY - cp22 Stunnel HTTP for mw121 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 5.378 second response time [22:28:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.60, 3.34, 2.88 [22:28:51] PROBLEM - cp22 Stunnel HTTP for mon141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:18] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:37] PROBLEM - cp22 Stunnel HTTP for phab121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:29:56] RECOVERY - cp22 Stunnel HTTP for test131 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 8.210 second response time [22:30:35] RECOVERY - cp22 Stunnel HTTP for mw122 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 6.649 second response time [22:30:45] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.45, 2.56, 2.64 [22:30:49] PROBLEM - cp22 Stunnel HTTP for puppet141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:48] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 2.00, 1.86, 1.72 [22:33:05] RECOVERY - cp22 Stunnel HTTP for mw142 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 3.803 second response time [22:34:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.41, 3.41 [22:35:08] PROBLEM - cp22 Stunnel HTTP for mw131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:29] PROBLEM - cp22 Stunnel HTTP for reports121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:09] RECOVERY - cp22 Stunnel HTTP for mw141 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 6.027 second response time [22:36:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.31, 3.37 [22:36:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.26, 1.69, 1.69 [22:38:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.36, 1.67 [22:38:07] PROBLEM - cp22 Stunnel HTTP for mail121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:38:33] PROBLEM - cp22 Stunnel HTTP for matomo131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:38:33] RECOVERY - cp22 Stunnel HTTP for mon141 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 36298 bytes in 0.760 second response time [22:38:37] RECOVERY - cp22 SSH on cp22 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [22:38:37] RECOVERY - cp22 PowerDNS Recursor on cp22 is OK: DNS OK: 4.008 seconds response time. miraheze.org returns 109.228.51.216,2a00:da00:1800:328::1 [22:38:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:39:14] RECOVERY - cp22 Stunnel HTTP for phab121 on cp22 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 6.270 second response time [22:39:27] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3361 bytes in 1.069 second response time [22:39:35] RECOVERY - cp22 Stunnel HTTP for mw131 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 3.312 second response time [22:39:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:40:02] RECOVERY - cp22 Stunnel HTTP for mw132 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 2.011 second response time [22:40:11] RECOVERY - cp22 Stunnel HTTP for reports121 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 8.736 second response time [22:40:29] RECOVERY - cp22 ferm_active on cp22 is OK: OK ferm input default policy is set [22:40:41] RECOVERY - cp22 Stunnel HTTP for puppet141 on cp22 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 9.197 second response time [22:40:55] RECOVERY - cp22 Stunnel HTTP for matomo131 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 8.173 second response time [22:42:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.00, 3.55 [22:42:47] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.41, 2.34, 1.93 [22:42:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [22:42:56] PROBLEM - cp22 SSH on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:44] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 217.174.247.33/cpweb, 2a00:da00:1800:326::1/cpweb [22:43:44] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:45] PROBLEM - cp22 PowerDNS Recursor on cp22 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:44:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.80, 3.54 [22:44:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.59, 1.79, 1.76 [22:44:07] RECOVERY - cp22 Stunnel HTTP for mwtask141 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 356 bytes in 8.493 second response time [22:44:47] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.05, 1.85, 1.80 [22:45:37] PROBLEM - cp22 Stunnel HTTP for reports121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 4.12, 3.69 [22:46:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.31, 2.31, 1.95 [22:46:21] PROBLEM - cp22 Stunnel HTTP for puppet141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:51] PROBLEM - cp22 Stunnel HTTP for mw122 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:44] PROBLEM - cp22 ferm_active on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:47:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.62, 3.56 [22:48:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.24, 1.79, 1.80 [22:49:20] PROBLEM - cp22 Stunnel HTTP for matomo131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:27] PROBLEM - cp22 Stunnel HTTP for mw132 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:47] PROBLEM - cp22 Stunnel HTTP for phab121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.35, 1.55, 1.70 [22:50:47] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.38, 1.56, 1.69 [22:51:10] PROBLEM - cp22 Stunnel HTTP for mw131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:13] PROBLEM - cp22 Stunnel HTTP for mw121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:23] PROBLEM - cp22 Stunnel HTTP for mw142 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:41] PROBLEM - cp22 Stunnel HTTP for mw141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:47] RECOVERY - cp22 Stunnel HTTP for mw122 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 9.352 second response time [22:51:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 2.88, 3.27 [22:52:06] PROBLEM - cp22 Stunnel HTTP for mon141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:58] PROBLEM - cp22 Stunnel HTTP for mwtask141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.52, 1.92, 1.79 [22:57:28] PROBLEM - cp22 Stunnel HTTP for mw122 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 4.57, 3.86 [22:58:57] RECOVERY - cp22 Stunnel HTTP for mw131 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 9.407 second response time [22:59:19] RECOVERY - cp22 Stunnel HTTP for mw141 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.590 second response time [23:00:24] PROBLEM - cp22 Stunnel HTTP for test131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:54] RECOVERY - cp22 Stunnel HTTP for mail121 on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 9.439 second response time [23:00:54] RECOVERY - cp22 Stunnel HTTP for reports121 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 9.492 second response time [23:01:54] RECOVERY - cp22 Stunnel HTTP for puppet141 on cp22 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 8.636 second response time [23:04:47] PROBLEM - cp22 Stunnel HTTP for mw131 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:08] PROBLEM - cp22 Stunnel HTTP for mw141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:23] RECOVERY - cp22 Stunnel HTTP for mw122 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 8.433 second response time [23:05:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.74, 3.82 [23:06:48] PROBLEM - cp22 Stunnel HTTP for mail121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:59] PROBLEM - cp22 Stunnel HTTP for reports121 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.87, 1.57, 1.91 [23:08:22] PROBLEM - cp22 Stunnel HTTP for puppet141 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:09:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.94, 3.86 [23:11:32] PROBLEM - cp22 Stunnel HTTP for mw122 on cp22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:13] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.47, 3.44, 2.40 [23:14:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.17, 1.62 [23:15:10] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.84, 2.60, 2.21 [23:16:09] PROBLEM - ping6 on cp22 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:16:51] PROBLEM - Host cp22 is DOWN: PING CRITICAL - Packet loss = 100% [23:19:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.66, 3.33, 3.82 [23:20:59] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.12, 5.17, 3.36 [23:22:56] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.20, 3.56, 2.99 [23:25:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.46, 3.30 [23:25:39] RECOVERY - Host cp22 is UP: PING OK - Packet loss = 0%, RTA = 6.74 ms [23:25:39] RECOVERY - cp22 Stunnel HTTP for mw131 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.028 second response time [23:25:40] RECOVERY - cp22 Stunnel HTTP for mw121 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.033 second response time [23:25:44] RECOVERY - cp22 ferm_active on cp22 is OK: OK ferm input default policy is set [23:25:45] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.79, 0.24, 0.08 [23:25:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:25:48] RECOVERY - cp22 Stunnel HTTP for puppet141 on cp22 is OK: HTTP OK: Status line output matched "403" - 289 bytes in 0.068 second response time [23:25:49] RECOVERY - cp22 conntrack_table_size on cp22 is OK: OK: nf_conntrack is 0 % full [23:25:50] RECOVERY - cp22 Stunnel HTTP for mw132 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.028 second response time [23:25:52] RECOVERY - cp22 Stunnel HTTP for mw142 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.358 second response time [23:25:53] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 11% [23:25:55] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [23:25:58] RECOVERY - cp22 Stunnel HTTP for reports121 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.113 second response time [23:26:02] RECOVERY - cp22 Stunnel HTTP for mwtask141 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 356 bytes in 0.156 second response time [23:26:06] RECOVERY - cp22 SSH on cp22 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:26:09] RECOVERY - cp22 Stunnel HTTP for matomo131 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 100182 bytes in 0.248 second response time [23:26:16] RECOVERY - cp22 Stunnel HTTP for phab121 on cp22 is OK: HTTP OK: Status line output matched "500" - 2855 bytes in 0.109 second response time [23:26:18] RECOVERY - cp22 Stunnel HTTP for test131 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 14740 bytes in 0.079 second response time [23:26:19] RECOVERY - cp22 Stunnel HTTP for mon141 on cp22 is OK: HTTP OK: HTTP/1.1 200 OK - 36298 bytes in 0.075 second response time [23:26:19] RECOVERY - cp22 Stunnel HTTP for mail121 on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.076 second response time [23:26:19] !log manually add ipv6 address to cp22 as it appears after a reboot it didn't get its address? Also rebooted because the cpu was at 100% [23:26:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:18] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.17, 3.72, 3.11 [23:27:28] RECOVERY - cp22 Stunnel HTTP for mw141 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.061 second response time [23:27:28] RECOVERY - cp22 PowerDNS Recursor on cp22 is OK: DNS OK: 0.028 seconds response time. miraheze.org returns 109.228.51.216,2a00:da00:1800:328::1 [23:27:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:27:33] RECOVERY - cp22 Stunnel HTTP for mw122 on cp22 is OK: HTTP OK: HTTP/1.1 204 No Content - 352 bytes in 0.023 second response time [23:27:33] RECOVERY - ping6 on cp22 is OK: PING OK - Packet loss = 0%, RTA = 6.62 ms [23:27:33] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3354 bytes in 0.032 second response time [23:28:01] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 2.44, 6.34, 3.55 [23:28:06] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:16] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.10, 3.86, 3.26 [23:31:14] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.17, 2.86, 2.96 [23:31:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.66, 3.59 [23:31:56] RECOVERY - mon141 Current Load on mon141 is OK: OK - load average: 0.58, 3.12, 2.84 [23:33:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.88, 5.17, 4.15 [23:41:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.90, 1.65 [23:43:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.88, 1.66 [23:43:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.22, 3.49, 2.93 [23:45:13] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.86, 1.69 [23:45:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.54, 3.01 [23:49:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.15, 2.98 [23:49:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 2.00, 1.86 [23:55:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 4.13, 2.36, 1.95 [23:55:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.45, 1.67