[00:00:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.93, 4.03, 3.47 [00:01:36] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.94, 4.22, 3.40 [00:05:41] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:05:43] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/060cea483581...9d3de84f4529 [00:05:44] [02miraheze/ImportDump] 07Universal-Omega 039d3de84 - implement functionality to retrieve interwiki prefixes. [00:05:51] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9d3de84f4529...78d31f919e42 [00:05:53] [02miraheze/ImportDump] 07Universal-Omega 0378d31f9 - Add ImportDumpInterwikiMap config [00:05:54] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:07:20] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.89, 1.94 [00:08:42] miraheze/ImportDump - Universal-Omega the build has errored. [00:09:27] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:09:28] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/78d31f919e42...e3febdeee19b [00:09:30] [02miraheze/ImportDump] 07Universal-Omega 03e3febde - Update ImportDumpRequestViewer.php [00:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e3febdeee19b...dd0f22ea788c [00:11:17] [02miraheze/ImportDump] 07Universal-Omega 03dd0f22e - Update ImportDumpRequestManager.php [00:11:19] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:12:13] miraheze/ImportDump - Universal-Omega the build has errored. [00:12:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.01, 3.48, 3.74 [00:13:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.40, 1.70 [00:13:29] miraheze/ImportDump - Universal-Omega the build has errored. [00:13:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [00:13:53] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:16:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:16:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.81, 4.64, 4.12 [00:18:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.87, 3.58, 3.80 [00:19:25] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/dd0f22ea788c...5d72e15fe5fd [00:19:27] [02miraheze/ImportDump] 07Universal-Omega 035d72e15 - Try to use `$wgInterwikiCentralDB` if there is none in local interwiki [00:19:28] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:21:58] miraheze/ImportDump - Universal-Omega the build has errored. [00:23:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.45, 3.84 [00:23:20] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5d72e15fe5fd...9935d092e2e3 [00:23:21] [02miraheze/ImportDump] 07Universal-Omega 039935d09 - Update ImportDumpRequestManager.php [00:23:23] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:24:37] PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ct777.cf All nameservers failed to answer the query. [00:25:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.49, 3.78 [00:25:34] miraheze/ImportDump - Universal-Omega the build has errored. [00:27:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.43, 3.71 [00:28:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.81, 3.71 [00:30:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.54, 3.54, 3.62 [00:31:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 2.76, 3.35 [00:32:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 3.62, 3.62 [00:34:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.29, 3.50 [00:42:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.16, 2.95, 3.33 [00:43:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:44:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.38, 3.97 [00:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:53:15] RECOVERY - wiki.ct777.cf - reverse DNS on sslhost is OK: SSL OK - wiki.ct777.cf reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:58:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:58:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.53, 3.62 [01:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:08:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.60, 3.85 [01:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:10:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 4.15, 4.01 [01:11:10] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9935d092e2e3...af63f70aca08 [01:11:12] [02miraheze/ImportDump] 07Universal-Omega 03af63f70 - Fully implement command generation [01:11:13] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:11:24] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/af63f70aca08...6056a4295f2b [01:11:25] [02miraheze/ImportDump] 07Universal-Omega 036056a42 - Add ImportDumpScriptCommand config [01:11:27] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:11:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.47, 1.76, 1.41 [01:12:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 2.26, 1.72 [01:13:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.09, 1.51, 1.36 [01:14:01] miraheze/ImportDump - Universal-Omega the build has errored. [01:14:42] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.84, 1.63 [01:16:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.69, 3.89 [01:16:38] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6056a4295f2b...9c9e800c551a [01:16:40] [02miraheze/ImportDump] 07Universal-Omega 039c9e800 - fix [01:16:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.64, 1.58 [01:16:41] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:17:54] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9c9e800c551a...e60b04e007d2 [01:17:56] [02miraheze/ImportDump] 07Universal-Omega 03e60b04e - fix [01:17:57] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:18:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.89, 3.93 [01:19:10] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:20:24] miraheze/ImportDump - Universal-Omega the build has errored. [01:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:22:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 1.96, 1.72 [01:22:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.46, 3.25 [01:22:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.54, 3.29 [01:24:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.95, 3.97 [01:24:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.35, 3.34, 3.23 [01:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:28:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.56, 1.67 [01:29:49] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e60b04e007d2...4b7ec56037a5 [01:29:50] [02miraheze/ImportDump] 07Universal-Omega 034b7ec56 - Update ServiceWiring.php [01:29:52] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:30:47] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 2.95, 3.19 [01:31:52] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4b7ec56037a5...032eef34712c [01:31:53] [02miraheze/ImportDump] 07Universal-Omega 03032eef3 - Update ImportDumpRequestManager.php [01:31:55] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:32:17] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/032eef34712c...4861731a6f89 [01:32:19] [02miraheze/ImportDump] 07Universal-Omega 034861731 - Update ImportDumpRequestViewer.php [01:32:20] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:32:26] miraheze/ImportDump - Universal-Omega the build has errored. [01:35:42] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4861731a6f89...edb423615fa9 [01:35:43] [02miraheze/ImportDump] 07Universal-Omega 03edb4236 - Update ImportDumpRequestManager.php [01:35:45] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:35:53] miraheze/ImportDump - Universal-Omega the build has errored. [01:36:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 2.94, 3.40 [01:39:04] miraheze/ImportDump - Universal-Omega the build has errored. [01:46:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.90, 3.51 [01:49:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.00, 2.06, 1.46 [01:51:29] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:51:31] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/edb423615fa9...16c7dcdc8384 [01:51:32] [02miraheze/ImportDump] 07Universal-Omega 0316c7dcd - Update ImportDumpRequestManager.php [01:51:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.81, 1.44 [01:53:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.96, 1.55, 1.39 [01:55:35] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:55:57] miraheze/ImportDump - Universal-Omega the build passed. [01:56:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 2.91, 3.25 [01:59:55] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 17.46, 12.63, 9.42 [02:00:35] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:47] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:54] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:53] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.986 second response time [02:02:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.122 second response time [02:02:42] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.310 second response time [02:02:50] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.325 second response time [02:06:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.32, 10.84, 9.84 [02:08:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.62, 9.64, 9.51 [02:12:28] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/16c7dcdc8384...3719553413c3 [02:12:30] [02miraheze/ImportDump] 07Universal-Omega 033719553 - Update ImportDumpRequestManager.php [02:12:31] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:16:19] miraheze/ImportDump - Universal-Omega the build passed. [02:26:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5adaa5615f7d [02:26:12] [02miraheze/mw-config] 07Universal-Omega 035adaa56 - Add ImportDump configs [02:26:13] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:26:15] [02mw-config] 07Universal-Omega opened pull request 03#4619: Add ImportDump configs - 13https://github.com/miraheze/mw-config/pull/4619 [02:27:25] miraheze/mw-config - Universal-Omega the build passed. [02:28:47] [02mw-config] 07Universal-Omega closed pull request 03#4619: Add ImportDump configs - 13https://github.com/miraheze/mw-config/pull/4619 [02:28:48] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:28:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/160a4629849b...a65830d82018 [02:28:51] [02miraheze/mw-config] 07Universal-Omega 03a65830d - Add ImportDump configs (#4619) [02:28:53] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:29:57] miraheze/mw-config - Universal-Omega the build passed. [02:41:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.23, 2.84 [02:43:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.21, 3.09, 2.83 [02:44:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.99, 3.33 [02:46:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.69, 3.29 [02:48:20] !log [@test101] starting deploy of {'config': True} to all [02:48:21] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [02:48:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:13] !log [@mwtask111] starting deploy of {'config': True} to all [02:52:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:21] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [02:52:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 4.29, 3.66 [02:52:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 4.00, 3.64 [02:56:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.96, 3.65 [02:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.62, 3.56 [03:03:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.67, 3.43, 2.18 [03:04:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.25, 3.97, 3.68 [03:06:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.95, 3.24, 3.45 [03:08:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.69, 3.58 [03:10:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.31, 3.46 [03:11:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.85, 1.81, 1.88 [03:14:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.62, 3.55 [03:16:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.79, 3.19, 3.39 [03:17:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.38, 1.46, 1.70 [03:17:44] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/3719553413c3...9f1e7c236ef8 [03:17:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.03, 2.51, 1.71 [03:17:45] [02miraheze/ImportDump] 07Universal-Omega 039f1e7c2 - Use str_replace [03:17:47] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:21:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.51, 1.54, 1.51 [03:22:06] miraheze/ImportDump - Universal-Omega the build passed. [03:24:52] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9f1e7c236ef8...aa9064996668 [03:24:54] [02miraheze/ImportDump] 07Universal-Omega 03aa90649 - Update ImportDumpRequestManager.php [03:24:55] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:25:18] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/aa9064996668...a4ea9e07cb25 [03:25:19] [02miraheze/ImportDump] 07Universal-Omega 03a4ea9e0 - Update ImportDumpRequestViewer.php [03:25:21] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:25:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.96, 4.82, 3.56 [03:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.36, 4.63, 3.75 [03:27:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 4.00, 3.22 [03:28:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.76, 3.54 [03:29:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.63, 3.12, 2.99 [03:29:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.57, 3.35 [03:29:52] miraheze/ImportDump - Universal-Omega the build passed. [03:30:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.25, 3.38 [03:31:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.84, 3.21, 3.23 [03:35:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a65830d82018...b81b03fcb257 [03:35:26] [02miraheze/mw-config] 07Universal-Omega 03b81b03f - Use ::class [03:36:26] miraheze/mw-config - Universal-Omega the build passed. [03:39:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b81b03fcb257...5f090ae3edc1 [03:39:55] [02miraheze/mw-config] 07Universal-Omega 035f090ae - Use ForeignDBViaLBRepo [03:41:00] miraheze/mw-config - Universal-Omega the build passed. [03:41:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.62, 3.44 [03:41:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.63, 3.27 [03:43:02] [02mw-config] 07Universal-Omega commented on pull request 03#4617: $wmgSharedUploadClientDBname for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4617#issuecomment-1105972424 [03:43:04] [02mw-config] 07Universal-Omega closed pull request 03#4617: $wmgSharedUploadClientDBname for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4617 [03:43:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.81, 3.53 [03:43:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.25, 3.36, 3.20 [03:47:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.41, 3.28 [03:47:14] [02mw-config] 07Universal-Omega reviewed pull request 03#4612 commit - 13https://github.com/miraheze/mw-config/pull/4612#discussion_r855744384 [03:48:05] !log [@test101] starting deploy of {'config': True} to all [03:48:06] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:48:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.95, 3.73 [03:49:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.17, 3.20 [03:50:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 2.13, 1.76 [03:52:02] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.48, 1.95, 1.75 [03:52:17] !log [@mwtask111] starting deploy of {'config': True} to all [03:52:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:24] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 6s [03:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.96, 3.97, 3.74 [03:55:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.45, 1.60 [03:56:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.55, 3.61 [03:58:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.67, 2.88, 3.35 [04:15:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [04:15:53] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:25:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.04, 1.73, 1.43 [04:27:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.61, 1.33, 1.31 [04:46:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.15, 2.86 [04:48:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 2.88, 2.78 [05:13:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.81, 1.35 [05:13:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [05:15:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.46, 1.37, 1.25 [05:30:46] [02mw-config] 07ugochimobi reviewed pull request 03#4612 commit - 13https://github.com/miraheze/mw-config/pull/4612#discussion_r855778563 [05:32:02] [02mw-config] 07Universal-Omega reviewed pull request 03#4612 commit - 13https://github.com/miraheze/mw-config/pull/4612#discussion_r855778997 [05:32:18] [02mw-config] 07Universal-Omega reviewed pull request 03#4612 commit - 13https://github.com/miraheze/mw-config/pull/4612#discussion_r855778997 [05:32:36] [02mw-config] 07Universal-Omega commented on pull request 03#4612: override page HTML title with the wikibase label, if available, or else the id - 13https://github.com/miraheze/mw-config/pull/4612#issuecomment-1106027112 [05:33:14] [02mw-config] 07ugochimobi commented on pull request 03#4617: $wmgSharedUploadClientDBname for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4617#issuecomment-1106027366 [05:33:20] [02mw-config] 07Universal-Omega reviewed pull request 03#4612 commit - 13https://github.com/miraheze/mw-config/pull/4612#discussion_r855779499 [05:34:04] [02mw-config] 07Universal-Omega commented on pull request 03#4617: $wmgSharedUploadClientDBname for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4617#issuecomment-1106027715 [05:35:10] [02mw-config] 07ugochimobi reviewed pull request 03#4612 commit - 13https://github.com/miraheze/mw-config/pull/4612#discussion_r855780171 [05:35:51] [02mw-config] 07ugochimobi commented on pull request 03#4612: override page HTML title with the wikibase label, if available, or else the id - 13https://github.com/miraheze/mw-config/pull/4612#issuecomment-1106028504 [05:37:58] [02mw-config] 07ugochimobi commented on pull request 03#4617: $wmgSharedUploadClientDBname for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4617#issuecomment-1106029360 [05:39:26] [02mw-config] 07Universal-Omega commented on pull request 03#4618: Update LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4618#issuecomment-1106029982 [05:43:54] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [05:43:54] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:53:07] [02mw-config] 07ugochimobi edited pull request 03#4618: Update LocalSettings.php with wgPageImagesExpandOpenSearchXml/Namespace for gratisdatawiki and gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4618 [05:53:50] [02mw-config] 07ugochimobi commented on pull request 03#4618: Update LocalSettings.php with wgPageImagesExpandOpenSearchXml/Namespace for gratisdatawiki and gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4618#issuecomment-1106036238 [05:57:02] [02mw-config] 07Universal-Omega edited pull request 03#4618: Set wgPageImagesExpandOpenSearchXml/Namespace for gratisdatawiki and gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4618 [05:57:12] PROBLEM - wiki.lu9.io - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lu9.io could not be found [05:57:39] [02mw-config] 07Universal-Omega closed pull request 03#4618: Set wgPageImagesExpandOpenSearchXml/Namespace for gratisdatawiki and gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4618 [05:57:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5f090ae3edc1...828cbdd3221b [05:57:43] [02miraheze/mw-config] 07ugochimobi 03828cbdd - Set wgPageImagesExpandOpenSearchXml/Namespace for gratisdatawiki and gpcommonswiki (#4618) [05:58:42] miraheze/mw-config - Universal-Omega the build passed. [06:01:07] [02mw-config] 07Universal-Omega closed pull request 03#4601: Re-enable CreateRedirect; Disable EditNotify - 13https://github.com/miraheze/mw-config/pull/4601 [06:01:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/828cbdd3221b...680c03b70ece [06:01:10] [02miraheze/mw-config] 07Universal-Omega 03680c03b - Re-enable CreateRedirect; Disable EditNotify (#4601) [06:01:11] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [06:01:13] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [06:02:20] miraheze/mw-config - Universal-Omega the build passed. [06:13:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:19:38] !log [@test101] starting deploy of {'config': True} to all [06:19:39] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [06:19:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:19:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:19:56] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.83, 1.23 [06:21:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.72, 1.26 [06:22:53] !log [@mwtask111] starting deploy of {'config': True} to all [06:22:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:04] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 10s [06:23:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:49] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.61, 1.28 [06:26:53] RECOVERY - wiki.lu9.io - reverse DNS on sslhost is OK: SSL OK - wiki.lu9.io reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:27:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.72, 1.39 [06:29:16] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855736630 [06:29:17] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855732435 [06:29:19] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855743757 [06:29:20] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855739886 [06:29:22] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855741883 [06:29:23] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855745605 [06:29:25] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855749134 [06:29:26] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855765029 [06:29:28] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855766837 [06:29:29] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855760679 [06:29:31] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855770766 [06:32:42] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855807162 [06:33:16] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855807475 [06:33:21] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855807533 [06:33:30] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.65, 1.47 [06:34:12] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855808003 [06:34:44] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855808292 [06:35:19] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855808607 [06:35:48] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855808845 [06:36:14] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855809090 [06:36:53] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855809456 [06:37:42] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855809905 [06:37:53] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855810010 [06:39:01] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855810656 [06:40:05] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855811459 [06:40:55] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855811947 [06:42:07] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855812645 [06:44:32] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855814027 [06:44:44] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855814135 [06:44:55] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855814250 [06:45:56] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855814863 [06:46:32] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855815260 [06:47:57] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855816127 [06:48:10] [02ImportDump] 07RhinosF1 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855816255 [06:49:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.91, 1.64 [06:51:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.78, 1.62 [06:52:13] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855818702 [06:52:22] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855818802 [06:53:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.26, 1.60, 1.58 [06:54:28] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855820165 [06:55:25] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855820834 [06:58:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.29, 2.75 [07:00:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 2.81, 2.64 [07:04:44] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855826665 [07:06:51] I'm sorry that there is no 'Mark this as resolved' buttons. I didn't expect it. [07:15:15] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855836383 [07:19:40] [02ImportDump] 07Universal-Omega reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855843890 [07:26:40] [02ImportDump] 07lens0021 reviewed pull request 03#1 commit - 13https://github.com/miraheze/ImportDump/pull/1#discussion_r855848701 [07:34:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.44, 2.82 [07:36:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.01, 2.74 [07:45:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [07:45:53] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:43:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:26:41] [02mw-config] 07ugochimobi opened pull request 03#4620: setting gpcommonswiki as a wgForeignFileRepos for gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4620 [09:26:59] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.41, 1.06 [09:27:46] miraheze/mw-config - ugochimobi the build passed. [09:28:28] [02mw-config] 07Universal-Omega closed pull request 03#4620: setting gpcommonswiki as a wgForeignFileRepos for gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4620 [09:28:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/680c03b70ece...45d157083695 [09:28:31] [02miraheze/mw-config] 07ugochimobi 0345d1570 - setting gpcommonswiki as a wgForeignFileRepos for gratispaideiawiki (#4620) [09:28:58] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.29, 1.45, 1.12 [09:29:41] miraheze/mw-config - Universal-Omega the build passed. [09:43:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:43:53] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:48:51] !log [@test101] starting deploy of {'config': True} to all [09:48:52] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [09:49:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:49:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:52:20] !log [@mwtask111] starting deploy of {'config': True} to all [09:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:52:31] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 11s [09:52:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:36] [02mw-config] 07ugochimobi opened pull request 03#4621: Update Wikibase.php with configuration for gratisdatawiki, gratispaideiawiki, gpcommonswiki, benpediawiki - 13https://github.com/miraheze/mw-config/pull/4621 [10:04:46] miraheze/mw-config - ugochimobi the build passed. [10:05:11] [02mw-config] 07ugochimobi synchronize pull request 03#4621: Update Wikibase.php with configuration for gratisdatawiki, gratispaideiawiki, gpcommonswiki, benpediawiki - 13https://github.com/miraheze/mw-config/pull/4621 [10:06:23] miraheze/mw-config - ugochimobi the build passed. [10:13:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:30:08] [02mw-config] 07ugochimobi opened pull request 03#4622: set wgLexemeLanguageCodePropertyId and wgLexemeEnableDataTransclusion for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4622 [10:31:18] miraheze/mw-config - ugochimobi the build passed. [10:43:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:15:19] [02mw-config] 07ugochimobi opened pull request 03#4623: set wgPropertySuggesterMinProbability for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4623 [11:16:23] miraheze/mw-config - ugochimobi the build passed. [11:26:13] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.73, 1.71, 1.19 [11:28:13] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.93, 1.32, 1.10 [11:38:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.56, 2.89 [11:42:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.17, 4.04, 3.20 [11:42:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.44, 2.86 [11:44:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.65, 3.16 [11:44:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.77, 3.08, 2.79 [11:48:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.05, 3.05 [11:59:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.60, 3.05, 2.96 [12:01:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.02, 2.73, 2.86 [12:06:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.16, 2.88 [12:08:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 2.73, 2.75 [12:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 4.06, 3.40 [12:28:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.34, 2.26, 1.66 [12:30:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.84, 3.11 [12:30:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.96, 3.28 [12:31:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.77, 3.07, 2.92 [12:32:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:32:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 3.26, 3.10 [12:32:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.65, 3.48 [12:32:53] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.03, 1.82, 1.63 [12:34:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.77, 3.53 [12:36:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.85, 1.45, 1.54 [12:38:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.85, 3.62 [12:42:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.73, 3.04, 3.37 [12:47:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:54:04] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [13:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:13:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:19:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 4.08, 3.36 [13:20:04] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:23:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.90, 3.46 [13:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:25:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.99, 4.73, 3.80 [13:26:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.48, 3.11 [13:30:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.92, 3.29, 3.13 [13:33:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.79, 3.76 [13:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:36:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.74, 3.73 [13:38:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.60, 3.29 [13:40:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.45, 3.27 [13:42:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.19, 3.20 [13:42:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.61, 3.68 [13:44:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 4.15, 3.86 [13:46:10] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.91, 10.51, 9.12 [13:46:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.89, 3.80 [13:47:43] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/python3/ [13:47:43] kages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407145738601685 seconds [13:48:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.35, 10.43, 9.27 [13:50:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.34, 9.86, 9.21 [13:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:56:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.76, 3.73 [14:01:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:02:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.97, 3.86 [14:03:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.64, 3.41 [14:04:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.64, 4.38, 4.01 [14:05:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.88, 4.33, 3.68 [14:08:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.88, 3.47 [14:09:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.76, 3.63 [14:14:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.77, 3.55 [14:15:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [14:15:53] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:16:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.10, 3.18, 3.36 [14:16:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.13, 3.62, 3.96 [14:17:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.12, 2.89, 3.33 [14:20:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.36, 9.96, 9.01 [14:23:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:26:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.15, 9.41, 9.14 [14:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.28, 3.84, 3.74 [14:29:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.54, 3.26 [14:30:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.81, 3.74 [14:31:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.50, 3.26 [14:32:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.05, 9.77, 8.92 [14:33:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.38, 9.68, 8.80 [14:33:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 3.10, 3.14 [14:35:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.40, 9.63, 8.90 [14:36:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.95, 8.94, 8.86 [14:36:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.67, 3.66 [14:37:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.29, 3.20 [14:38:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.80, 3.72 [14:39:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.65, 3.19, 3.17 [14:42:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 3.96, 3.75 [14:43:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:58] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.80, 1.83, 1.49 [14:44:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.82, 3.72 [14:45:43] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:47:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.24, 1.87, 1.58 [14:48:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.98, 3.79 [14:51:55] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.26, 1.75, 1.61 [14:54:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.78, 3.78 [14:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:55:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.19, 1.98, 1.73 [14:59:13] !log db101: SET GLOBAL table_open_cache=50000; [14:59:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:35] it seems it was 16119 even tho in the config we have 50000? [14:59:52] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.05, 1.78, 1.74 [15:00:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 2.99, 3.39 [15:01:51] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.38, 1.65, 1.70 [15:01:53] !log db111: SET GLOBAL table_open_cache=50000; [15:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:02:21] !log db121: SET GLOBAL table_open_cache=50000; [15:02:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:05:46] !log db112: SET GLOBAL table_open_cache=50000; [15:05:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:06:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.52, 3.47 [15:10:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.01, 3.29 [15:11:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0b8d39079152 [15:11:05] [02miraheze/puppet] 07paladox 030b8d390 - prometheus: increase scrape timeout for blackbox jobs [15:11:07] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [15:11:08] [02puppet] 07paladox opened pull request 03#2515: prometheus: increase scrape timeout for blackbox jobs - 13https://github.com/miraheze/puppet/pull/2515 [15:13:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [15:17:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.44, 3.39 [15:18:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.70, 11.02, 9.37 [15:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:19:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.15, 1.43, 1.05 [15:19:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.76, 3.50 [15:20:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.75, 10.99, 9.56 [15:20:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.01, 9.55, 8.75 [15:21:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.65, 1.45, 1.10 [15:21:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.57, 3.47 [15:22:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.73, 9.14, 8.69 [15:22:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7615806cf078 [15:22:43] [02miraheze/puppet] 07paladox 037615806 - prometheus: fix url for blackbox [15:22:44] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [15:22:45] [02puppet] 07paladox opened pull request 03#2516: prometheus: fix url for blackbox - 13https://github.com/miraheze/puppet/pull/2516 [15:22:53] PROBLEM - wiki.fatcatfablab.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fatcatfablab.org' expires in 15 day(s) (Sun 08 May 2022 15:10:49 GMT +0000). [15:23:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.77, 3.54 [15:24:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.26, 9.35, 9.25 [15:26:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.48, 3.50 [15:28:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/18e2158427db...105b6cf2b222 [15:28:23] [02miraheze/ssl] 07MirahezeSSLBot 03105b6cf - Bot: Update SSL cert for wiki.fatcatfablab.org [15:28:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.18, 3.38 [15:31:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f5d9fb38e548 [15:31:06] [02miraheze/puppet] 07paladox 03f5d9fb3 - varnish: add mw*_test without probe or pool [15:31:07] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [15:31:09] [02puppet] 07paladox opened pull request 03#2517: varnish: add mw*_test without probe or pool - 13https://github.com/miraheze/puppet/pull/2517 [15:31:15] PROBLEM - sharqedia.ml - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sharqedia.ml' expires in 15 day(s) (Sun 08 May 2022 15:20:12 GMT +0000). [15:32:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.64, 3.27 [15:32:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 4.05, 3.70 [15:33:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/105b6cf2b222...f5252ed9d3fc [15:33:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.85, 3.40 [15:33:22] [02miraheze/ssl] 07MirahezeSSLBot 03f5252ed - Bot: Update SSL cert for sharqedia.ml [15:33:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:34:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.94, 4.08, 3.49 [15:34:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.94, 3.71 [15:36:13] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.84, 1.53, 1.29 [15:36:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.49, 3.58, 3.38 [15:36:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 4.19, 3.82 [15:37:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.53, 3.41 [15:38:16] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 1.65, 1.36 [15:38:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 3.04, 3.21 [15:38:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.70, 3.68 [15:39:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.25, 3.32 [15:40:13] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.84, 1.37, 1.30 [15:42:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.87, 3.74 [15:44:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.73, 3.71 [15:45:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [15:45:53] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:46:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.44, 4.32, 3.92 [15:47:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.52, 3.35 [15:47:30] PROBLEM - forecasting.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'forecasting.wiki' expires in 15 day(s) (Sun 08 May 2022 15:22:02 GMT +0000). [15:48:37] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:50:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.61, 10.52, 9.27 [15:51:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.35, 3.32 [15:52:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.07, 9.42, 9.01 [15:52:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.77, 3.84 [15:53:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f5252ed9d3fc...69fa5ab27dfa [15:53:16] [02miraheze/ssl] 07MirahezeSSLBot 0369fa5ab - Bot: Update SSL cert for forecasting.wiki [15:53:37] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:54:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/54145c91e456...cfaf4d5a4f5c [15:54:42] [02miraheze/puppet] 07paladox 03cfaf4d5 - Update mediawiki.conf [15:57:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.54, 3.23 [15:59:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cfaf4d5a4f5c...d0236afc7111 [15:59:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.09, 3.09 [15:59:45] [02miraheze/puppet] 07paladox 03d0236af - mediawiki: allow prometheus server to connect to port 80/443 [16:00:29] RECOVERY - sharqedia.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'sharqedia.ml' will expire on Thu 21 Jul 2022 14:33:15 GMT +0000. [16:02:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.18, 2.99, 3.40 [16:02:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.24, 9.53 [16:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:03:33] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.62, 1.43 [16:04:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.51, 10.36, 9.66 [16:05:32] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.79, 1.33, 1.35 [16:06:45] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.39, 2.20, 1.60 [16:08:40] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.58, 1.64, 1.46 [16:10:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.10, 10.56, 9.87 [16:10:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.53, 9.87, 9.74 [16:11:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.34, 10.82, 10.08 [16:14:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 10.56, 10.06 [16:14:59] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.61, 3.22 [16:15:49] [02puppet] 07JohnFLewis commented on pull request 03#2517: varnish: add mw*_test without probe or pool - 13https://github.com/miraheze/puppet/pull/2517#issuecomment-1106665089 [16:15:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.93, 10.03, 9.99 [16:16:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.06, 4.37, 3.66 [16:16:29] RECOVERY - forecasting.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'forecasting.wiki' will expire on Thu 21 Jul 2022 14:53:09 GMT +0000. [16:17:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.46, 10.52 [16:19:40] [02puppet] 07paladox commented on pull request 03#2517: varnish: add mw*_test without probe or pool - 13https://github.com/miraheze/puppet/pull/2517#issuecomment-1106668157 [16:19:48] [02puppet] 07paladox closed pull request 03#2517: varnish: add mw*_test without probe or pool - 13https://github.com/miraheze/puppet/pull/2517 [16:19:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.45, 10.63 [16:19:50] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [16:19:51] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:20:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.79, 3.50 [16:21:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.85, 11.53, 10.39 [16:21:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.78, 10.85 [16:22:38] RECOVERY - wiki.fatcatfablab.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fatcatfablab.org' will expire on Thu 21 Jul 2022 14:28:14 GMT +0000. [16:22:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.21, 11.41, 10.48 [16:23:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.88, 10.42 [16:23:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.82, 11.82, 10.98 [16:24:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6f11c1ff9423 [16:24:19] [02miraheze/puppet] 07paladox 036f11c1f - nginx: redo keep alive [16:24:20] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:24:22] [02puppet] 07paladox opened pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:24:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.63, 4.02, 3.69 [16:24:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.66, 10.49, 9.49 [16:25:24] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.97, 1.47, 1.24 [16:25:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6f11c1ff9423...6a0de6c268ee [16:25:51] [02miraheze/puppet] 07paladox 036a0de6c - Update init.pp [16:25:53] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:26:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.27, 11.04, 10.47 [16:26:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.49, 10.17, 9.50 [16:26:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.58, 3.57 [16:27:24] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.88, 2.52, 1.66 [16:27:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.52, 9.86, 10.19 [16:27:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.69, 3.73 [16:28:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.62, 10.20, 10.24 [16:29:23] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.17, 1.99, 1.57 [16:29:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.69, 3.71 [16:30:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6a0de6c268ee...1bb44ffee21a [16:30:39] [02miraheze/puppet] 07paladox 031bb44ff - Update mw101.yaml [16:30:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.93, 9.74, 10.06 [16:30:41] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:31:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1bb44ffee21a...12759cbe5f7d [16:31:02] [02miraheze/puppet] 07paladox 0312759cb - Update mw102.yaml [16:31:03] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:31:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/12759cbe5f7d...6f7bc10706a6 [16:31:14] [02miraheze/puppet] 07paladox 036f7bc10 - Update mw111.yaml [16:31:15] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:31:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.50, 1.47, 1.43 [16:31:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6f7bc10706a6...6f706204463b [16:31:28] [02miraheze/puppet] 07paladox 036f70620 - Update mw112.yaml [16:31:29] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:31:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6f706204463b...109aa9a6fecb [16:31:45] [02miraheze/puppet] 07paladox 03109aa9a - Update mw121.yaml [16:31:46] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:31:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/109aa9a6fecb...ad239068e645 [16:31:57] [02miraheze/puppet] 07paladox 03ad23906 - Update mw122.yaml [16:31:59] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:33:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.95, 3.87 [16:33:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ad239068e645...2fc77551cf5c [16:33:38] [02miraheze/puppet] 07paladox 032fc7755 - Update test101.yaml [16:33:40] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:34:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2fc77551cf5c...b6dfa6621417 [16:34:13] [02miraheze/puppet] 07paladox 03b6dfa66 - Update mwtask111.yaml [16:34:15] [02puppet] 07paladox synchronize pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:34:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.93, 2.96, 3.32 [16:35:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.01, 10.67 [16:36:22] [02puppet] 07paladox closed pull request 03#2518: nginx: redo keep alive - 13https://github.com/miraheze/puppet/pull/2518 [16:36:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±10] 13https://github.com/miraheze/puppet/compare/d0236afc7111...9491fb8b1a9b [16:36:25] [02miraheze/puppet] 07paladox 039491fb8 - nginx: redo keep alive (#2518) [16:36:26] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:36:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [16:37:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 10.20, 10.41 [16:39:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 4.14, 3.91 [16:41:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.75, 3.79 [16:41:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.06, 8.83, 9.81 [16:43:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9491fb8b1a9b...4ba30e26d60d [16:43:02] [02miraheze/puppet] 07paladox 034ba30e2 - prometheus: don't set preferred_ip_protocol to ipv4 in blackbox [16:43:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 3.67, 3.74 [16:43:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:45:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.91, 4.45, 3.43 [16:45:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.56, 3.33 [16:47:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.85, 3.33 [16:47:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 3.03, 3.16 [16:49:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.79, 3.36 [16:49:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 10.50, 10.15 [16:51:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.59, 3.32 [16:53:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.67, 3.95 [16:53:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.37, 2.72, 3.03 [16:53:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 11.49, 10.63 [16:57:07] Hi SRE [16:57:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.17, 10.94, 10.66 [16:59:22] I want to suggest a change to the way UploadWizard is currently being configured in LocalSettings.php, I don't know if I'm correct though, lol, yeah, but let me drop it. [17:00:43] This is the way it currently is... https://www.irccloud.com/pastebin/YqEy4ilp/LS.php [17:02:48] I don't see the possibility of having a per-wiki configuration with this though? [17:02:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 3.95, 3.83 [17:03:10] that's the main reason for me suggestion [17:03:21] s/me/my [17:03:21] ugochimobi meant to say: that's the main reason for my suggestion [17:03:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.70, 3.40 [17:05:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.44, 10.34, 9.56 [17:05:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 11.16, 10.81 [17:06:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.44, 3.66 [17:07:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.30, 3.30 [17:07:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.60, 9.26, 9.26 [17:07:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 10.32, 10.55 [17:12:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.59, 3.64 [17:12:58] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.38, 3.20 [17:13:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.94, 9.52, 10.10 [17:15:03] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.92, 1.86, 1.55 [17:16:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.81, 3.74 [17:17:02] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.67, 1.52 [17:17:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.51, 10.61, 10.46 [17:18:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.21, 3.23 [17:19:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 11.40, 10.76 [17:20:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.93, 11.05, 9.54 [17:20:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 4.34, 3.93 [17:21:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 11.10, 10.75 [17:22:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.86, 10.33, 9.34 [17:22:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.85, 3.79 [17:23:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.79, 10.77, 9.78 [17:23:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.00, 11.64, 10.21 [17:24:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.66, 9.21, 9.04 [17:25:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 11.45, 10.89 [17:26:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.95, 11.04, 9.62 [17:26:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.83, 3.80 [17:27:00] Pardon me if this seems very dumb :) https://www.irccloud.com/pastebin/WYMuJjwz/Amended [17:27:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.52, 9.93, 9.65 [17:27:47] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.62, 9.23, 9.57 [17:27:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.22, 10.79, 10.70 [17:28:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.56, 3.28 [17:28:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.39, 11.55, 9.96 [17:28:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.79, 3.79 [17:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:28:57] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.20, 1.80, 1.66 [17:30:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.46, 3.28 [17:30:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 3.85, 3.80 [17:30:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.39, 1.30, 1.49 [17:31:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.36, 9.52, 10.17 [17:32:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 3.87, 3.45 [17:32:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.09, 11.98, 10.63 [17:32:23] CosmicAlpha: Reception123: ^ [17:34:04] [02mw-config] 07ugochimobi opened pull request 03#4624: add UploadWizard Flickr configs for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4624 [17:34:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.81, 3.82 [17:35:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.83, 11.47, 10.14 [17:35:14] miraheze/mw-config - ugochimobi the build passed. [17:35:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.09, 10.47, 9.89 [17:35:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.92, 11.52, 10.85 [17:36:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.88, 3.30, 3.31 [17:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:37:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.51, 11.55, 10.35 [17:37:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.33, 9.48, 9.58 [17:37:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 11.17, 10.79 [17:38:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.62, 9.44, 9.98 [17:41:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.28, 8.81, 9.51 [17:43:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.10, 9.30, 10.16 [17:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 3.78, 3.66 [17:47:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.15, 3.86, 3.43 [17:49:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.91, 3.51 [17:49:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 10.25, 10.29 [17:49:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 3.62, 3.40 [17:51:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 3.14, 3.25 [17:51:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.96, 11.08, 10.58 [17:52:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.75, 3.83 [17:54:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 3.18, 3.30 [17:55:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.93, 11.40, 10.89 [18:00:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 4.51, 4.11 [18:00:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.48, 3.40 [18:02:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.06, 3.23, 3.34 [18:04:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.16, 3.56, 3.84 [18:04:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.30, 10.51, 9.51 [18:08:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.39, 4.12 [18:08:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.09, 9.70, 9.44 [18:10:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.76, 1.40 [18:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:12:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.46, 1.64, 1.40 [18:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:19:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.84, 1.18 [18:20:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.39, 9.44, 8.95 [18:21:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 1.95, 1.31 [18:22:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.15, 9.80, 9.15 [18:22:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.54, 3.32, 3.89 [18:23:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 11.32, 10.80 [18:24:37] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.89, 1.71, 1.51 [18:24:48] [02puppet] 07paladox closed pull request 03#2515: prometheus: increase scrape timeout for blackbox jobs - 13https://github.com/miraheze/puppet/pull/2515 [18:24:50] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [18:24:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [18:25:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.33, 1.63, 1.33 [18:25:04] [02puppet] 07paladox closed pull request 03#2516: prometheus: fix url for blackbox - 13https://github.com/miraheze/puppet/pull/2516 [18:26:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.33, 11.20, 9.81 [18:26:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.61, 1.63, 1.51 [18:28:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.97, 11.11, 9.96 [18:28:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.60, 3.42, 3.71 [18:29:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.88, 11.25, 10.96 [18:33:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 10.96, 10.87 [18:34:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.29, 9.33, 9.59 [18:34:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.93, 3.87 [18:35:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 11.14, 10.95 [18:36:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.51, 4.57, 4.11 [18:36:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:47] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.87, 3.50 [18:37:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 11.55, 11.11 [18:38:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.59, 3.18 [18:38:21] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.54, 3.42 [18:39:56] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.77, 1.55 [18:40:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 3.17, 3.08 [18:40:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.338 second response time [18:40:50] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.320 second response time [18:40:53] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.023 second response time [18:40:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.021 second response time [18:40:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 3.711 second response time [18:41:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.95, 3.80, 3.51 [18:41:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.41, 11.25, 11.16 [18:42:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.72, 3.95 [18:43:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.58, 3.45 [18:43:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.84, 1.62 [18:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 4.04, 4.04 [18:45:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.73, 3.31, 3.37 [18:45:43] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.53, 1.69, 1.59 [18:45:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:46:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.42, 3.82 [18:48:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.48, 3.77 [18:48:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:49:37] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.91, 1.70 [18:51:33] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.88, 1.72 [18:52:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.38, 3.70 [18:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:56:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.31, 3.55 [18:57:21] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.47, 1.64, 1.67 [18:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.21, 3.48 [18:59:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.25, 3.03 [19:00:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.44, 3.35 [19:01:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:02:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.30, 3.32 [19:03:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.38, 3.12 [19:05:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.38, 3.14 [19:07:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.00, 3.03 [19:09:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.75, 9.34, 10.14 [19:10:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 3.63, 3.56 [19:11:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.99, 3.58 [19:12:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 10.30, 9.72 [19:14:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.36, 9.55, 9.51 [19:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:15:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.05, 10.52, 9.41 [19:17:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.71, 3.59 [19:19:25] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.82, 10.13, 9.50 [19:20:10] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.34, 10.80, 10.13 [19:20:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.91, 3.70 [19:21:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.90, 1.67 [19:21:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 10.66, 10.34 [19:22:04] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [19:23:02] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.81, 1.67 [19:23:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.87, 11.28, 10.59 [19:24:05] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.47, 9.37, 9.76 [19:25:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.65, 1.62 [19:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:25:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.48, 10.62, 10.44 [19:27:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.15, 3.36 [19:27:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.04, 10.60 [19:27:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.63, 11.53, 10.52 [19:28:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 2.98, 3.36 [19:29:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.52, 10.40, 9.76 [19:29:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 11.00, 10.64 [19:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:31:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.73, 9.48, 9.50 [19:31:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 11.10, 10.70 [19:31:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.89, 11.73, 10.86 [19:34:18] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:47] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:55] Hmm [19:36:15] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.858 second response time [19:36:43] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.323 second response time [19:36:45] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.018 second response time [19:36:46] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 0.615 second response time [19:36:52] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.969 second response time [19:36:57] Not sure why it locked up [19:37:00] Cc CosmicAlpha [19:37:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.75, 9.27, 10.09 [19:39:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 11.58, 11.41 [19:41:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.53, 10.17, 10.24 [19:43:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:44:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.57, 3.42 [19:45:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.85, 9.76, 10.10 [19:46:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.87, 11.72, 10.12 [19:46:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.91, 3.20, 3.32 [19:48:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.60, 11.04, 10.05 [19:50:04] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:52:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.33, 9.99, 9.85 [19:59:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:59:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 10.96, 10.74 [20:01:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 11.12, 10.83 [20:06:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.33, 4.26, 3.46 [20:08:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.77, 2.12, 1.46 [20:10:11] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.59, 1.86, 1.44 [20:11:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.25, 10.97, 10.75 [20:12:07] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 2.29, 1.66 [20:12:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.50, 3.63, 3.49 [20:13:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 10.06, 10.42 [20:14:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.88, 1.76, 1.54 [20:14:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.85, 1.54 [20:14:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 4.16, 3.69 [20:15:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.57, 3.12 [20:15:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:16:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.38, 1.66, 1.53 [20:16:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.58, 1.47 [20:17:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 3.81, 3.23 [20:17:49] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.60, 9.53, 10.13 [20:19:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.76, 3.27 [20:19:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.66, 3.27 [20:21:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.78, 3.33 [20:22:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.75, 1.57 [20:23:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 4.09, 3.53 [20:24:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:24:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.48, 1.62, 1.54 [20:25:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.87, 3.49 [20:25:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.72, 3.48 [20:25:50] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.45, 2.39, 1.67 [20:27:50] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.90, 1.78, 1.53 [20:28:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.65, 3.94 [20:29:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.90, 3.54 [20:29:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.17, 4.09, 3.64 [20:29:49] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.87, 1.43, 1.43 [20:30:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.96, 4.02 [20:32:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.37, 9.32, 8.65 [20:34:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.29, 8.30, 8.34 [20:35:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.92, 3.74 [20:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:37:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.78, 3.74 [20:39:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 4.70, 4.10 [20:39:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.80, 4.55, 4.00 [20:40:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.16, 10.53, 8.74 [20:40:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 10.74, 10.22 [20:41:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.05, 10.10, 8.84 [20:42:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 10.40, 8.94 [20:42:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 10.53, 10.19 [20:43:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.60, 9.59, 8.80 [20:43:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.69, 10.22, 9.33 [20:44:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.62, 9.95, 10.03 [20:44:41] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.07, 9.78, 8.90 [20:45:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.03, 3.91, 3.92 [20:45:28] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.77, 9.82, 9.29 [20:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:47:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.81, 3.97 [20:53:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 2.53, 3.26 [20:54:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.31, 3.83 [20:55:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 2.52, 3.30 [20:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:56:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.12, 3.66, 3.87 [20:57:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.55, 3.54 [20:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.39, 3.74 [21:01:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:02:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.42, 3.42 [21:02:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 3.97, 3.86 [21:04:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.12, 3.31 [21:08:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.78, 3.83 [21:10:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 4.11, 3.94 [21:11:14] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.55, 1.34 [21:13:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.47, 1.56, 1.37 [21:13:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [21:15:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.95, 3.71 [21:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:17:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.73, 3.65 [21:19:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.83, 3.70 [21:21:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.49, 3.59 [21:24:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.40, 3.83 [21:24:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.81, 2.71, 3.25 [21:25:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:26:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.70, 3.88 [21:28:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.37, 3.23 [21:28:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 3.98, 3.66 [21:30:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 3.05, 3.14 [21:30:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.67, 3.86 [21:30:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.24, 3.42 [21:32:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:32:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.53, 3.50 [21:33:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.68, 1.50 [21:34:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 3.10, 3.35 [21:34:55] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.45, 4.33, 2.50 [21:35:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.24, 1.47, 1.44 [21:37:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 10.79, 9.92 [21:40:40] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.36, 1.67, 1.85 [21:42:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:42:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.77, 3.29 [21:42:40] [02dns] 07MacFan4000 opened pull request 03#269: add wikicompliance.com zone - 13https://github.com/miraheze/dns/pull/269 [21:44:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.87, 1.17, 1.60 [21:46:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.76, 9.65, 9.77 [21:47:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:52:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:56:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 10.38, 10.10 [22:00:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.08, 9.25, 9.76 [22:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:04:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.89, 3.45 [22:07:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.46, 12.29, 10.71 [22:08:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.58, 3.06 [22:08:18] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:47] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:49] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 5.012 second response time [22:10:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.88, 3.66 [22:10:44] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.432 second response time [22:10:46] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.982 second response time [22:10:46] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.129 second response time [22:10:48] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.971 second response time [22:11:49] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/5acac305d1c6 [22:11:51] [02miraheze/ssl] 07MacFan4000 035acac30 - add 4 domains [22:11:52] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [22:11:54] [02ssl] 07MacFan4000 opened pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [22:11:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.97, 2.94 [22:12:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.50, 4.02, 3.75 [22:13:31] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a4ea9e07cb25...153e0b350c29 [22:13:32] [02miraheze/ImportDump] 07Universal-Omega 03153e0b3 - Update ImportDumpRequestViewer.php [22:13:34] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:13:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 11.21, 10.92 [22:13:53] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/153e0b350c29...16a1690133a7 [22:13:54] [02miraheze/ImportDump] 07Universal-Omega 0316a1690 - Update en.json [22:13:56] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:14:12] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/16a1690133a7...4bdb4ca9e827 [22:14:13] [02miraheze/ImportDump] 07Universal-Omega 034bdb4ca - Update qqq.json [22:14:15] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:14:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.62, 3.63 [22:14:42] [02ssl] 07MacFan4000 closed pull request 03#505: Add redirects - 13https://github.com/miraheze/ssl/pull/505 [22:14:44] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/69fa5ab27dfa...dc20f9bfbdea [22:14:45] [02miraheze/ssl] 07AgentIsai 03dc20f9b - Add redirects (#505) [22:16:27] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4bdb4ca9e827...ee9a849e4953 [22:16:29] [02miraheze/ImportDump] 07Universal-Omega 03ee9a849 - Update SpecialRequestImportDump.php [22:16:30] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:18:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.74, 4.08, 3.78 [22:18:59] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/ee9a849e4953...9cd2f17e97ac [22:19:01] [02miraheze/ImportDump] 07Universal-Omega 039cd2f17 - Update includes/ImportDumpRequestQueuePager.php [22:19:02] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [22:20:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.80, 3.71 [22:22:47] miraheze/ImportDump - Universal-Omega the build passed. [22:23:20] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/5acac305d1c6...83dd61bea126 [22:23:21] [02miraheze/ssl] 07MacFan4000 0383dd61b - add pmwiki.cubestudios.xyz [22:23:23] [02ssl] 07MacFan4000 synchronize pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [22:23:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.13, 8.82, 9.92 [22:24:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 1.93, 1.60 [22:25:50] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/83dd61bea126...2e71966db267 [22:25:51] [02miraheze/ssl] 07MacFan4000 032e71966 - add wiki.cjgh.xyz [22:25:52] [02ssl] 07MacFan4000 synchronize pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [22:26:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.61, 1.52 [22:27:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 10.41, 10.28 [22:30:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:30:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.61, 9.74, 10.03 [22:36:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.06, 3.35 [22:44:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.64, 1.45 [22:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:46:20] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.11, 1.95, 1.60 [22:48:18] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.62, 1.51 [22:48:45] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.triplescripts.org All nameservers failed to answer the query. [23:00:01] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:11:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.19, 3.03 [23:13:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.29, 4.54, 3.55 [23:15:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.77, 3.39 [23:17:41] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [23:17:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.36, 3.29 [23:19:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 10.40, 9.56 [23:20:37] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9cd2f17e97ac...54e7c2d2d709 [23:20:38] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.51, 2.01, 1.27 [23:20:39] [02miraheze/ImportDump] 07Universal-Omega 0354e7c2d - Update extension.json [23:20:40] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:22:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.95, 1.34 [23:22:43] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1360s [23:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:33] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1471s [23:24:36] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.39, 1.35, 1.19 [23:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:44] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/54e7c2d2d709...a0796bd3c222 [23:24:46] [02miraheze/ImportDump] 07Universal-Omega 03a0796bd - Update en.json [23:24:47] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:24:50] miraheze/ImportDump - Universal-Omega the build passed. [23:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:27:09] miraheze/ImportDump - Universal-Omega the build has errored. [23:27:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 3.80, 3.25 [23:27:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.85, 3.48 [23:29:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 4.38, 3.53 [23:29:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.55, 3.22 [23:29:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.50, 3.40 [23:31:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.40, 3.38 [23:33:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.94, 3.55 [23:33:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.26, 3.36, 3.22 [23:34:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.82, 11.24, 10.44 [23:35:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.89, 3.57 [23:36:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.58, 10.54, 10.28 [23:37:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.16, 4.84, 3.82 [23:38:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.42, 9.79, 10.03 [23:38:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.97, 3.66 [23:39:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.79, 3.65 [23:39:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.81, 3.97, 3.62 [23:40:34] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a0796bd3c222...658fd3ffbb43 [23:40:36] [02miraheze/ImportDump] 07Universal-Omega 03658fd3f - Update qqq.json [23:40:37] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:40:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.77, 4.23, 3.77 [23:41:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.24, 4.17, 3.79 [23:41:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:41:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 4.10, 3.71 [23:42:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.77, 3.66 [23:42:43] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [23:43:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.95, 3.75 [23:43:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.70, 3.61 [23:44:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.26, 4.38, 3.88 [23:45:02] miraheze/ImportDump - Universal-Omega the build passed. [23:49:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 2.90, 3.33 [23:51:56] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/2e71966db267...136972a6d741 [23:51:57] [02miraheze/ssl] 07MacFan4000 03136972a - add wiki.recaptime.eu.org [23:51:59] [02ssl] 07MacFan4000 synchronize pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [23:52:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.48, 1.30 [23:52:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 10.40, 9.90 [23:54:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.48, 1.33 [23:55:49] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/136972a6d741...bb46abdfa691 [23:55:50] [02miraheze/ssl] 07MacFan4000 03bb46abd - Update certs.yaml [23:55:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.37, 4.40, 3.83 [23:55:52] [02ssl] 07MacFan4000 synchronize pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [23:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:56:18] [02ssl] 07MacFan4000 synchronize pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [23:56:20] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bb46abdfa691...4f89155af105 [23:56:21] [02miraheze/ssl] 07MacFan4000 034f89155 - Update certs.yaml [23:56:48] [02ssl] 07MacFan4000 closed pull request 03#506: add 4 domains - 13https://github.com/miraheze/ssl/pull/506 [23:56:50] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [23:56:51] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [23:56:53] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+4/-0/±1] 13https://github.com/miraheze/ssl/compare/dc20f9bfbdea...7d7c24f8622a [23:56:54] [02miraheze/ssl] 07MacFan4000 037d7c24f - add 4 domains (#506) [23:57:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.66, 3.56 [23:57:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.16, 10.06, 9.12 [23:59:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.47, 9.69, 9.11