[00:01:17] paladox: yeah I was. deploy-mediawiki with rebuild i18n. [00:01:25] ohhh i see [00:01:41] CosmicAlpha: i restarted test101 as i saw puppet was running since almost an hour before [00:02:02] oh... [00:02:03] and i saw high load so restarted but looking at the graph it looked like something was running all day [00:02:14] was trying to deploy the ldap change [00:02:21] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.24, 3.34, 2.54 [00:02:24] test101 only has 1 core :/ [00:03:07] yeah, it's OK, and yes, it does not have enough resources to run so much... it is running redis, jobrunner, and pretty much everything each mw server does with not even close to the same resources. [00:03:34] sorry that i restarted CosmicAlpha [00:04:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.70, 1.38 [00:04:20] I don't know why it is even set up as a redis server, I do think I said before that it would not have the resources to run all that. But it was set as a redis server when I was away for 2 months. Oh well. And it is fine paladox, no worries at all about restarting it. [00:04:20] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.73, 3.05, 2.53 [00:04:40] ok [00:05:04] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True} to all [00:05:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.31, 1.27 [00:08:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.24, 6.22, 5.35 [00:12:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.00, 5.93, 5.48 [00:14:41] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 577s [00:15:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:55] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:26:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.09, 3.90 [00:26:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 6.55, 5.92 [00:28:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 3.57, 3.96 [00:28:09] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.21, 6.23, 5.87 [00:30:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.11, 3.75 [00:32:34] [02PDFEmbed] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/PDFEmbed [00:32:34] [url] Page not found · GitHub · GitHub | github.com [00:32:35] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/PDFEmbed/commit/43800025d62d [00:32:36] [02miraheze/PDFEmbed] 07Universal-Omega 034380002 - Delete PDFHandler.class.php [00:32:38] [02PDFEmbed] 07Universal-Omega opened pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:32:38] [url] Page not found · GitHub · GitHub | github.com [00:33:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.24, 2.90, 3.85 [00:33:31] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/43800025d62d...0ef55ec49eb0 [00:33:31] [url] Comparing 43800025d62d...0ef55ec49eb0 · miraheze/PDFEmbed · GitHub | github.com [00:33:32] [02miraheze/PDFEmbed] 07Universal-Omega 030ef55ec - Update extension.json [00:33:34] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:33:34] [url] Page not found · GitHub · GitHub | github.com [00:34:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.39, 2.54, 3.36 [00:36:44] miraheze/PDFEmbed - Universal-Omega the build passed. [00:36:58] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/0ef55ec49eb0...a4c08c6b07e6 [00:36:59] [url] Comparing 0ef55ec49eb0...a4c08c6b07e6 · miraheze/PDFEmbed · GitHub | github.com [00:37:00] [02miraheze/PDFEmbed] 07Universal-Omega 03a4c08c6 - Update PDFEmbed.hooks.php [00:37:01] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:37:01] [url] Page not found · GitHub · GitHub | github.com [00:38:43] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/PDFEmbed/compare/a4c08c6b07e6...335e84acfcc4 [00:38:43] [url] Comparing a4c08c6b07e6...335e84acfcc4 · miraheze/PDFEmbed · GitHub | github.com [00:38:45] [02miraheze/PDFEmbed] 07Universal-Omega 03335e84a - Create composer.json [00:38:46] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:38:46] ... [00:39:24] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/PDFEmbed/compare/335e84acfcc4...7888a39b7aa5 [00:39:24] [url] Comparing 335e84acfcc4...7888a39b7aa5 · miraheze/PDFEmbed · GitHub | github.com [00:39:25] [02miraheze/PDFEmbed] 07Universal-Omega 037888a39 - Create dependabot.yml [00:39:27] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:39:27] [url] Page not found · GitHub · GitHub | github.com [00:39:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 2.91, 3.97 [00:40:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.48, 1.83, 1.62 [00:41:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 2.54, 3.30 [00:41:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.30, 3.98 [00:41:54] miraheze/PDFEmbed - Universal-Omega the build has errored. [00:43:10] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/7888a39b7aa5...5f43d85595f1 [00:43:10] [url] Comparing 7888a39b7aa5...5f43d85595f1 · miraheze/PDFEmbed · GitHub | github.com [00:43:11] [02miraheze/PDFEmbed] 07Universal-Omega 035f43d85 - Update .phpcs.xml [00:43:13] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:43:13] [url] Page not found · GitHub · GitHub | github.com [00:44:29] [02miraheze/PDFEmbed] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/5f43d85595f1...22cd1fad106e [00:44:30] [url] Comparing 5f43d85595f1...22cd1fad106e · miraheze/PDFEmbed · GitHub | github.com [00:44:31] [02miraheze/PDFEmbed] 07github-actions 0322cd1fa - CI: lint code to MediaWiki standards [00:44:32] [02PDFEmbed] 07github-actions[bot] synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:44:33] [url] Page not found · GitHub · GitHub | github.com [00:45:34] miraheze/PDFEmbed - Universal-Omega the build has errored. [00:46:45] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.75, 1.72 [00:48:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.55, 1.66 [00:49:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 4.14, 3.76 [00:49:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.59, 4.05, 3.57 [00:50:01] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Delete PDFHandler.class.php - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:50:01] [url] Page not found · GitHub · GitHub | github.com [00:50:02] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/22cd1fad106e...27b1c388800c [00:50:03] [url] Comparing 22cd1fad106e...27b1c388800c · miraheze/PDFEmbed · GitHub | github.com [00:50:04] [02miraheze/PDFEmbed] 07Universal-Omega 0327b1c38 - Update PDFEmbed.hooks.php [00:50:14] [02PDFEmbed] 07Universal-Omega edited pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:50:14] [url] Page not found · GitHub · GitHub | github.com [00:50:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.63, 1.79, 1.12 [00:50:55] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:51:15] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:51:16] ... [00:51:17] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/PDFEmbed/compare/27b1c388800c...5e29d29ebe74 [00:51:17] [url] Comparing 27b1c388800c...5e29d29ebe74 · miraheze/PDFEmbed · GitHub | github.com [00:51:18] [02miraheze/PDFEmbed] 07Universal-Omega 035e29d29 - Update and rename PDFEmbed.hooks.php to includes/PDFEmbedHooks.php [00:51:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 4.00, 3.62 [00:52:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.56, 1.12 [00:52:28] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:52:28] [url] Page not found · GitHub · GitHub | github.com [00:52:29] miraheze/PDFEmbed - Universal-Omega the build has errored. [00:52:30] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/5e29d29ebe74...f29da45932d1 [00:52:30] [url] Comparing 5e29d29ebe74...f29da45932d1 · miraheze/PDFEmbed · GitHub | github.com [00:52:31] [02miraheze/PDFEmbed] 07Universal-Omega 03f29da45 - Update extension.json [00:52:43] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:52:43] [url] Page not found · GitHub · GitHub | github.com [00:52:45] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/f29da45932d1...9922857c6c25 [00:52:45] [url] Comparing f29da45932d1...9922857c6c25 · miraheze/PDFEmbed · GitHub | github.com [00:52:46] [02miraheze/PDFEmbed] 07Universal-Omega 039922857 - Update composer.json [00:53:01] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:53:01] [url] Page not found · GitHub · GitHub | github.com [00:53:02] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/9922857c6c25...e1490238a86a [00:53:02] [url] Comparing 9922857c6c25...e1490238a86a · miraheze/PDFEmbed · GitHub | github.com [00:53:04] [02miraheze/PDFEmbed] 07Universal-Omega 03e149023 - Update PDFEmbedHooks.php [00:53:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.96, 4.76, 3.94 [00:54:03] miraheze/PDFEmbed - Universal-Omega the build has errored. [00:54:31] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/PDFEmbed/compare/e1490238a86a...03991fe7219e [00:54:31] [url] Comparing e1490238a86a...03991fe7219e · miraheze/PDFEmbed · GitHub | github.com [00:54:32] [02miraheze/PDFEmbed] 07Universal-Omega 0303991fe - Create config.php [00:54:34] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [00:54:34] [url] Page not found · GitHub · GitHub | github.com [00:57:32] miraheze/PDFEmbed - Universal-Omega the build has errored. [01:03:28] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [01:03:28] [url] Page not found · GitHub · GitHub | github.com [01:03:29] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/03991fe7219e...acf26369f1b5 [01:03:30] [url] Comparing 03991fe7219e...acf26369f1b5 · miraheze/PDFEmbed · GitHub | github.com [01:03:31] [02miraheze/PDFEmbed] 07Universal-Omega 03acf2636 - Update PDFEmbedHooks.php [01:06:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 9.58, 8.64 [01:06:29] miraheze/PDFEmbed - Universal-Omega the build has errored. [01:08:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.89, 8.99, 8.54 [01:09:00] [02PDFEmbed] 07Universal-Omega synchronize pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [01:09:00] [url] Page not found · GitHub · GitHub | github.com [01:09:02] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/acf26369f1b5...14bc2ac8540c [01:09:02] [url] Comparing acf26369f1b5...14bc2ac8540c · miraheze/PDFEmbed · GitHub | github.com [01:09:03] [02miraheze/PDFEmbed] 07Universal-Omega 0314bc2ac - Update PDFEmbedHooks.php [01:12:37] miraheze/PDFEmbed - Universal-Omega the build passed. [01:12:51] [02PDFEmbed] 07Universal-Omega closed pull request 03#10: Cleanup and support 1.38 - 13https://github.com/miraheze/PDFEmbed/pull/10 [01:12:51] [url] Page not found · GitHub · GitHub | github.com [01:12:52] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+4/-2/±2] 13https://github.com/miraheze/PDFEmbed/compare/3d699c9e25e3...4c99f0d6485e [01:12:53] [url] Comparing 3d699c9e25e3...4c99f0d6485e · miraheze/PDFEmbed · GitHub | github.com [01:12:54] [02miraheze/PDFEmbed] 07Universal-Omega 034c99f0d - Cleanup and support 1.38 (#10) [01:12:55] [02PDFEmbed] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/PDFEmbed [01:12:55] [url] Page not found · GitHub · GitHub | github.com [01:12:57] [02miraheze/PDFEmbed] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:15:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7d06ede3970f...a82a6f02d93c [01:15:49] [url] Comparing 7d06ede3970f...a82a6f02d93c · miraheze/mediawiki · GitHub | github.com [01:15:50] [02miraheze/mediawiki] 07Universal-Omega 03a82a6f0 - Update PDFEmbed [01:16:02] !log [universalomega@test101] starting deploy of {'world': True} to all [01:16:02] miraheze/PDFEmbed - Universal-Omega the build passed. [01:16:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.80, 1.95, 1.71 [01:20:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.47, 1.66, 1.63 [01:20:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.74, 2.06, 1.30 [01:20:55] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 293s [01:21:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.32, 1.93, 1.36 [01:24:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.97, 1.74, 1.72 [01:24:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.22, 1.32, 1.21 [01:26:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.56, 1.65 [01:32:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.71, 2.04, 1.83 [01:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.43, 1.77, 1.36 [01:34:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.69, 1.73 [01:36:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.54, 1.67 [01:37:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [01:37:15] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:38:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.82, 1.56 [01:42:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.48, 6.41, 5.56 [01:42:22] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:44:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.00, 6.29, 5.63 [01:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.27, 2.32, 1.86 [01:58:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.38, 1.87, 1.96 [02:04:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.92, 1.92 [02:05:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.20, 6.22, 3.58 [02:06:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.13, 7.18, 6.62 [02:06:12] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [02:08:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.90, 6.66, 6.50 [02:10:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.36, 3.39, 3.96 [02:12:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.70, 4.00 [02:12:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.90, 7.13, 6.77 [02:14:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.00, 6.72, 6.66 [02:21:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.67, 1.18, 1.96 [02:23:49] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.35, 1.68, 1.10 [02:25:46] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.33, 1.49, 1.09 [02:26:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.18, 3.42, 3.87 [02:27:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.05, 1.66 [02:28:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 9.57, 8.32 [02:30:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.37, 9.14, 8.31 [02:32:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.97, 3.94 [02:35:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.57, 4.00 [02:35:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [02:35:10] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:36:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.91, 3.55, 3.78 [02:42:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.98, 3.85 [02:46:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.39, 3.96, 3.89 [02:47:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.50, 3.65 [02:49:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 2.98, 3.43 [02:51:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 2.86, 3.33 [02:56:24] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.43, 7.17, 6.60 [02:58:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.62, 3.64 [02:59:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 4.15, 3.73 [03:00:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.06, 6.30, 6.40 [03:04:08] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:04:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.61, 3.61 [03:06:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 4.03, 3.76 [03:07:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.96, 3.83 [03:08:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.70, 3.42, 3.56 [03:10:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.71, 3.64 [03:12:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.40, 3.53 [03:13:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.65, 2.56, 3.24 [03:14:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.62, 3.59 [03:14:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.44, 1.54, 1.95 [03:16:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.14, 3.42 [03:17:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.19, 3.83 [03:17:59] [02ImportDump] 07Universal-Omega closed pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:18:00] [url] Page not found · GitHub · GitHub | github.com [03:18:01] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [03:18:02] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+34/-0/±0] 13https://github.com/miraheze/ImportDump/compare/fd0601fe4712...3afef625cc29 [03:18:03] [url] Comparing fd0601fe4712...3afef625cc29 · miraheze/ImportDump · GitHub | github.com [03:18:04] [02miraheze/ImportDump] 07Universal-Omega 033afef62 - Create extension [03:18:05] [02ImportDump] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ImportDump [03:18:06] [url] Page not found · GitHub · GitHub | github.com [03:18:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.47, 2.85, 3.27 [03:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.34, 2.25, 2.12 [03:18:19] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-39.0.0 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/673fea1f91a2 [03:18:21] [02miraheze/ImportDump] 07dependabot[bot] 03673fea1 - Update mediawiki/mediawiki-codesniffer requirement from 38.0.0 to 39.0.0 [03:18:22] [02ImportDump] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-39.0.0 - 13https://github.com/miraheze/ImportDump [03:18:23] [url] Page not found · GitHub · GitHub | github.com [03:18:24] [02ImportDump] 07dependabot[bot] opened pull request 03#2: Update mediawiki/mediawiki-codesniffer requirement from 38.0.0 to 39.0.0 - 13https://github.com/miraheze/ImportDump/pull/2 [03:18:24] ... [03:18:25] [02ImportDump] 07dependabot[bot] labeled pull request 03#2: Update mediawiki/mediawiki-codesniffer requirement from 38.0.0 to 39.0.0 - 13https://github.com/miraheze/ImportDump/pull/2 [03:18:26] [url] Page not found · GitHub · GitHub | github.com [03:18:27] [02ImportDump] 07dependabot[bot] labeled pull request 03#2: Update mediawiki/mediawiki-codesniffer requirement from 38.0.0 to 39.0.0 - 13https://github.com/miraheze/ImportDump/pull/2 [03:18:27] ... [03:18:30] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/0af5d42a6bb4 [03:18:32] [02miraheze/ImportDump] 07dependabot[bot] 030af5d42 - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 [03:18:33] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/ImportDump [03:18:33] [url] Page not found · GitHub · GitHub | github.com [03:18:35] [02ImportDump] 07dependabot[bot] opened pull request 03#3: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ImportDump/pull/3 [03:18:35] ... [03:18:36] [02ImportDump] 07dependabot[bot] labeled pull request 03#3: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ImportDump/pull/3 [03:18:36] [url] Page not found · GitHub · GitHub | github.com [03:18:38] [02ImportDump] 07dependabot[bot] labeled pull request 03#3: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ImportDump/pull/3 [03:19:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.58, 3.89 [03:20:11] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/3afef625cc29...a5e4b260e878 [03:20:11] [url] Comparing 3afef625cc29...a5e4b260e878 · miraheze/ImportDump · GitHub | github.com [03:20:12] [02miraheze/ImportDump] 07dependabot[bot] 03a5e4b26 - Update mediawiki/mediawiki-codesniffer requirement from 38.0.0 to 39.0.0 (#2) [03:20:14] [02ImportDump] 07Universal-Omega closed pull request 03#2: Update mediawiki/mediawiki-codesniffer requirement from 38.0.0 to 39.0.0 - 13https://github.com/miraheze/ImportDump/pull/2 [03:20:14] [url] Page not found · GitHub · GitHub | github.com [03:20:15] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-39.0.0 - 13https://github.com/miraheze/ImportDump [03:20:16] [url] Page not found · GitHub · GitHub | github.com [03:20:17] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-39.0.0 [03:20:20] [02ImportDump] 07Universal-Omega closed pull request 03#3: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ImportDump/pull/3 [03:20:20] ... [03:20:22] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/a5e4b260e878...9984b91ae0cb [03:20:22] [url] Comparing a5e4b260e878...9984b91ae0cb · miraheze/ImportDump · GitHub | github.com [03:20:23] [02miraheze/ImportDump] 07dependabot[bot] 039984b91 - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 (#3) [03:20:25] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [03:20:26] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/ImportDump [03:20:26] [url] Page not found · GitHub · GitHub | github.com [03:21:15] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/652efd40febf...17ab72e264a7 [03:21:16] [url] Comparing 652efd40febf...17ab72e264a7 · miraheze/.github · GitHub | github.com [03:21:17] [02miraheze/.github] 07Universal-Omega 0317ab72e - Add ImportDump [03:21:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.15, 3.70 [03:22:31] [02.github] 07Universal-Omega closed pull request 03#11: Bump actions/cache from 3.0.3 to 3.0.4 in /workflow-templates - 13https://github.com/miraheze/.github/pull/11 [03:22:31] [url] Page not found · GitHub · GitHub | github.com [03:22:32] [02.github] 07dependabot[bot] commented on pull request 03#11: Bump actions/cache from 3.0.3 to 3.0.4 in /workflow-templates - 13https://github.com/miraheze/.github/pull/11#issuecomment-1149412201 [03:22:33] [url] Bump actions/cache from 3.0.3 to 3.0.4 in /workflow-templates by dependabot[bot] · Pull Request #11 · miraheze/.github · GitHub | github.com [03:22:34] [02.github] 07Universal-Omega deleted branch 03dependabot/github_actions/workflow-templates/actions/cache-3.0.4 - 13https://github.com/miraheze/.github [03:22:35] [02miraheze/.github] 07Universal-Omega deleted branch 03dependabot/github_actions/workflow-templates/actions/cache-3.0.4 [03:22:54] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.65, 2.14, 1.56 [03:24:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.90, 1.54 [03:25:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.90, 4.35, 4.04 [03:26:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.45, 3.40 [03:28:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.61, 3.45 [03:28:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.74, 1.96 [03:28:43] miraheze/ImportDump - dependabot[bot] the build passed. [03:28:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.91, 1.43, 1.44 [03:30:37] miraheze/ImportDump - dependabot[bot] the build passed. [03:31:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.99, 3.96 [03:32:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.08, 3.27 [03:33:10] miraheze/ImportDump - Universal-Omega the build passed. [03:35:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 4.15, 3.98 [03:35:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.95, 3.58 [03:37:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.82, 3.88 [03:40:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 1.82, 1.83 [03:41:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.59, 3.74 [03:41:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.08, 3.31, 3.40 [03:42:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.76, 1.81 [03:43:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.40, 3.64 [03:45:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.64, 3.69 [03:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 2.00, 1.86 [03:51:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.62, 3.71 [03:55:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.24, 7.27, 6.79 [03:58:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.80, 1.91 [03:59:01] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.74, 6.72, 6.67 [04:01:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.87, 3.16, 3.40 [04:02:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.47, 7.00, 6.82 [04:04:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.22, 2.12, 1.64 [04:05:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.79, 3.68 [04:06:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.64, 1.60, 1.50 [04:06:48] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.92, 6.69, 6.77 [04:07:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.89, 3.71 [04:09:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.71, 3.68 [04:10:31] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.83, 2.04, 1.70 [04:12:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.09, 1.72, 1.62 [04:14:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.96, 1.87 [04:14:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.80, 1.43, 1.53 [04:15:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.72, 3.62 [04:23:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 4.00, 3.86 [04:29:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 3.94, 3.84 [04:32:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.58, 3.24 [04:32:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.30, 3.45, 3.12 [04:34:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.53, 3.03, 3.09 [04:34:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 3.05, 3.02 [04:37:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.56, 3.81 [04:39:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.76, 3.84 [04:40:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.73, 1.96 [04:41:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.66, 3.79 [04:43:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.18, 3.05 [04:43:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.45, 5.25, 4.34 [04:43:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.23, 3.70, 3.23 [04:45:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.43, 3.19 [04:46:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 1.82, 1.91 [04:47:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.18, 3.12 [04:48:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.24, 3.19 [04:51:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.79, 3.50, 3.94 [04:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.98, 2.00 [04:59:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.29, 3.62 [05:01:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 2.87, 3.43 [05:03:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 2.81, 3.33 [05:10:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.53, 1.69 [05:16:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.71, 1.72 [05:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 2.08, 1.85 [05:21:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.44, 3.34 [05:22:29] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.38, 6.76, 5.96 [05:23:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.03, 3.19 [05:26:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.93, 1.92 [05:26:21] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.17, 6.60, 6.07 [05:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.75, 1.82 [05:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.74, 1.81 [05:42:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.84, 6.97, 6.50 [05:44:52] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.90, 6.05, 6.23 [05:45:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.79, 3.87, 3.09 [05:45:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 4.13, 3.43 [05:46:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 1.89, 1.80 [05:47:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 3.06, 2.88 [05:47:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.53, 3.30 [05:49:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.85, 2.94, 3.11 [05:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 2.00, 1.95 [06:00:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.96, 1.93 [06:20:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.26, 2.73 [06:20:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.97, 3.34 [06:22:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.52, 2.91, 2.66 [06:22:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 3.40, 3.20 [06:28:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 2.99, 2.62 [06:29:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.66, 2.99 [06:30:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.43, 2.83 [06:30:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.53, 3.23 [06:31:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.33, 2.95 [06:32:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.34, 3.20 [06:32:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.10, 2.78 [06:57:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.04, 3.43, 3.09 [06:57:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 4.02, 3.37 [06:59:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.97, 3.49, 3.14 [06:59:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.59, 3.18, 3.15 [07:01:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.10, 3.03 [07:11:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.58, 3.29 [07:13:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.25, 4.17, 3.54 [07:15:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.59, 3.41 [07:17:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.33, 3.33 [07:20:27] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.76, 1.57, 1.32 [07:24:23] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.50, 1.84, 1.46 [07:29:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a82a6f02d93c...3931602f7bf7 [07:29:35] [url] Comparing a82a6f02d93c...3931602f7bf7 · miraheze/mediawiki · GitHub | github.com [07:29:36] [02miraheze/mediawiki] 07Universal-Omega 033931602 - Update Chameleon [07:30:07] !log [universalomega@test101] starting deploy of {'world': True} to all [07:30:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:30:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.78, 1.91, 1.64 [07:32:13] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.22, 1.66, 1.58 [07:35:37] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 330s [07:35:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:45:08] !log [universalomega@test101] starting deploy of {'folders': 'w/skins/chameleon'} to all [07:45:10] !log [universalomega@test101] finished deploy of {'folders': 'w/skins/chameleon'} to all - SUCCESS in 1s [07:45:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:45:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:45:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.16, 3.01 [07:51:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.71, 3.08 [07:52:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.50, 2.94 [07:53:44] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.31, 3.00 [07:54:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 3.39, 2.96 [07:55:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.82, 3.65 [07:56:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/aa49bd46b865...646fd4b92972 [07:56:12] [url] Comparing aa49bd46b865...646fd4b92972 · miraheze/mw-config · GitHub | github.com [07:56:13] [02miraheze/mw-config] 07Universal-Omega 03646fd4b - Add Header Footer to site notice [07:57:25] miraheze/mw-config - Universal-Omega the build passed. [07:59:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/646fd4b92972...7d153127d7ae [07:59:39] [url] Comparing 646fd4b92972...7d153127d7ae · miraheze/mw-config · GitHub | github.com [07:59:40] [02miraheze/mw-config] 07Universal-Omega 037d15312 - HeaderFooter will be removed [08:00:44] miraheze/mw-config - Universal-Omega the build passed. [08:02:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.66, 3.29 [08:03:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 2.83, 3.26 [08:04:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.30, 3.20 [08:07:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.76, 3.56 [08:09:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.92, 3.05, 3.32 [08:22:21] !log [@mwtask111] starting deploy of {'config': True} to all [08:22:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:22:33] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 12s [08:22:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:22] !log [@test101] starting deploy of {'config': True} to all [08:24:23] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [08:24:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:28:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.15, 2.89 [08:32:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.88, 3.05, 2.90 [08:47:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.63, 1.84, 1.63 [08:49:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.74, 3.36 [08:53:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.57, 1.65, 1.63 [08:53:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 3.00, 3.16 [08:57:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.79, 1.70 [09:03:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.54, 1.63 [09:07:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.90, 1.76 [09:09:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.39, 1.78, 1.74 [09:11:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.32, 1.60, 1.68 [09:13:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.63, 3.06 [09:15:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.89, 1.78 [09:15:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.45, 3.06 [09:17:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.57, 4.19, 3.35 [09:19:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.26, 2.01, 1.86 [09:19:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.68, 3.26 [09:21:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.10, 3.09 [09:23:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.70, 1.88, 1.85 [09:25:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.13, 1.91, 1.86 [09:27:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.68, 1.88, 1.85 [09:34:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.63, 3.28 [09:36:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.10, 3.11 [09:37:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.11, 1.74, 1.74 [09:39:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.55, 1.68 [09:49:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 5.68, 2.87, 2.08 [09:57:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.22, 1.78, 1.86 [10:03:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.42, 1.67 [10:04:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.70, 1.97 [10:16:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.64, 1.75 [10:24:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.98, 1.95 [10:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.80, 1.84 [10:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.70, 1.80 [10:35:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.91, 1.48, 1.28 [10:36:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:37:23] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.54, 1.56, 1.34 [10:44:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.52, 1.68 [10:47:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 3.60, 3.10 [10:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.84, 2.12, 1.88 [10:48:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 2.88, 2.58 [10:49:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.37, 3.06 [10:50:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.65, 2.39, 2.43 [10:51:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.17, 2.02, 1.69 [10:51:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.15, 3.01 [10:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.94, 1.90 [10:56:11] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.01, 1.74, 1.38 [10:57:03] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.56, 7.01, 6.65 [10:58:07] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.20, 1.60, 1.37 [10:59:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 1.92, 1.84 [11:01:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 1.49, 1.69 [11:02:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.30, 1.80, 1.82 [11:02:52] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.83, 6.15, 6.43 [11:04:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.68, 1.78 [11:05:02] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [11:05:02] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:07:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.39, 2.23, 2.03 [11:09:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.86, 3.42 [11:11:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.27, 1.83, 1.91 [11:12:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.99, 1.87 [11:13:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 2.03, 1.98 [11:15:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.80, 1.88, 1.93 [11:15:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.41, 3.00 [11:17:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.13, 2.94, 2.87 [11:21:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.12, 1.95, 1.92 [11:21:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 9.79, 8.78 [11:22:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.87, 3.88 [11:23:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.42, 1.76, 1.85 [11:26:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 3.92, 3.86 [11:27:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.64, 9.67, 9.16 [11:28:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.46, 3.70 [11:29:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.60, 1.97, 1.88 [11:31:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.98, 1.95, 1.88 [11:31:55] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.60, 1.78, 1.49 [11:33:50] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.16, 1.55, 1.44 [11:35:44] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.66, 3.25 [11:35:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.86, 10.61, 9.62 [11:37:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 3.04, 3.07 [11:37:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.66, 9.75, 9.43 [11:38:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.83, 1.99 [11:39:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.93, 1.38, 1.66 [11:41:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.66, 3.64 [11:43:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.75, 4.04, 3.45 [11:43:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.12, 1.74, 1.53 [11:43:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.58, 3.61 [11:45:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.55, 3.34 [11:45:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.99, 1.79, 1.57 [11:47:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.66, 1.66 [11:47:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 3.17, 3.21 [11:47:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.10, 1.49, 1.48 [11:47:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.90, 3.73 [11:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.44, 2.02, 1.97 [11:49:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.85, 3.74 [11:51:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.04, 1.45, 1.59 [11:55:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.50, 3.66, 3.60 [11:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.81, 1.95 [11:57:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.96, 1.66, 1.62 [11:57:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.50, 1.42 [11:57:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.39, 3.51 [11:59:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.05, 2.23, 1.83 [11:59:09] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.04, 7.60, 6.62 [11:59:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.06, 1.26, 1.34 [11:59:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.04, 2.92, 3.32 [11:59:55] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 7.06, 9.05, 5.64 [12:02:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.90, 1.93 [12:03:01] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.49, 6.83, 6.56 [12:03:28] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:05:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.75, 1.76, 1.81 [12:06:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.84, 1.91 [12:12:27] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.284 second response time [12:12:27] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 3.443 second response time [12:12:28] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 3.273 second response time [12:12:42] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.32, 7.82, 7.08 [12:13:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.80, 1.79 [12:13:05] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [12:14:22] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.378 second response time [12:14:24] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.730 second response time [12:14:25] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.328 second response time [12:15:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.17, 1.65, 1.75 [12:15:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.34, 3.18 [12:17:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.49, 1.27, 1.59 [12:17:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.47, 3.24 [12:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 2.00, 1.90 [12:19:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.49, 3.28 [12:20:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 3.73, 3.13 [12:21:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.36, 3.28 [12:22:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.38, 3.07 [12:23:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 9.92, 8.79 [12:25:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.01, 2.13, 1.81 [12:26:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.95, 1.96 [12:27:52] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 1.76, 1.71 [12:27:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.53, 8.89 [12:29:50] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.15, 1.64, 1.68 [12:30:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.83, 1.90 [12:30:33] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.10, 1.85, 1.48 [12:31:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.52, 3.32 [12:33:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.14, 3.20 [12:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.88, 1.91 [12:34:24] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.25, 1.84, 1.58 [12:36:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.66, 1.44, 1.46 [12:37:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.89, 3.49 [12:37:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.57, 1.81, 1.73 [12:39:04] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [12:39:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.60, 3.43 [12:39:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.74, 2.07, 1.83 [12:41:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.28, 2.99 [12:42:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 3.58, 3.26 [12:42:36] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 1.97, 2.69, 3.78 [12:43:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.42, 3.07 [12:43:28] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.35, 1.90, 1.82 [12:43:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 4.16, 3.70 [12:44:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 3.27, 3.19 [12:45:38] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:47:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.23, 3.08 [12:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.83, 2.17, 1.97 [12:52:31] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.67, 2.63, 3.33 [12:53:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.62, 3.73 [12:55:14] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.79, 2.21, 1.96 [12:57:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.66, 3.37 [12:59:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.67, 3.32, 3.28 [13:00:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.80, 6.57, 7.83 [13:00:38] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:03:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.97, 4.07, 3.89 [13:03:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.39, 3.31 [13:05:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.88, 3.95, 3.52 [13:06:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.47, 3.31 [13:08:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.22, 3.23 [13:12:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.58, 5.35, 6.66 [13:13:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.03, 1.56, 1.94 [13:13:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.81, 3.67 [13:14:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.69, 3.43 [13:15:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 1.67, 1.93 [13:17:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.71, 1.77, 1.94 [13:19:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 9.88, 9.27 [13:19:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 4.26, 3.86 [13:21:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.41, 2.12, 2.04 [13:21:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.09, 9.79, 9.30 [13:21:57] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 2.46, 1.56 [13:23:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.29, 1.79, 1.92 [13:23:51] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.78, 1.83, 1.43 [13:25:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.40, 2.11, 2.03 [13:25:46] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.60, 1.40 [13:34:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.01, 10.08, 8.95 [13:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.77, 1.98 [13:35:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [13:35:28] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:35:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.68, 9.65, 8.94 [13:38:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.53, 9.64 [13:40:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.98, 10.27, 9.65 [13:42:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.92, 10.01, 9.62 [13:44:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.88, 1.92 [13:45:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.76, 1.78, 1.98 [13:45:37] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [13:47:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.23, 1.85, 1.97 [13:49:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.94, 1.83, 1.94 [13:52:01] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.28, 3.63, 2.95 [13:53:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.03, 2.24, 2.04 [13:54:00] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.83, 3.68, 3.05 [13:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.97, 1.99 [13:55:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.56, 1.91, 1.95 [13:55:38] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:59:26] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.29, 1.86, 1.41 [13:59:57] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.07, 3.38, 3.12 [14:01:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.07, 1.62, 1.39 [14:03:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.50, 2.18, 2.04 [14:04:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:05:53] PROBLEM - wiki.wisrail.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wisrail.de' expires in 15 day(s) (Fri 24 Jun 2022 13:34:23 GMT +0000). [14:07:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.51, 1.93, 1.97 [14:08:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.50, 1.69 [14:09:55] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.84, 7.62, 6.78 [14:11:51] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.11, 7.17, 6.73 [14:13:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.41, 1.36, 1.69 [14:13:47] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.11, 6.44, 6.51 [14:15:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.96, 1.68, 1.43 [14:16:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.38, 1.81, 1.72 [14:17:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.77, 1.72, 1.77 [14:17:26] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.21, 2.09, 1.60 [14:17:42] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 12.95, 9.03, 7.47 [14:19:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 2.03, 1.88 [14:19:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.52, 1.93, 1.61 [14:21:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.23, 1.63, 1.54 [14:22:14] PROBLEM - wiki.lu9.io - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lu9.io' expires in 15 day(s) (Fri 24 Jun 2022 14:09:45 GMT +0000). [14:23:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.39, 1.83, 1.85 [14:23:30] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.22, 7.98, 7.49 [14:25:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.08, 2.34, 2.03 [14:27:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.30, 2.00, 1.78 [14:28:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.82, 1.90 [14:31:26] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.50, 1.67, 1.69 [14:37:04] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.42, 5.77, 6.60 [14:38:12] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:38] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:39] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 2.02, 1.90 [14:40:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15887 bytes in 4.350 second response time [14:40:43] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15893 bytes in 8.895 second response time [14:40:58] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.52, 6.67, 6.76 [14:42:18] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.561 second response time [14:42:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.40, 6.54, 6.70 [14:52:27] PROBLEM - wildbowiki.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wildbowiki.net' expires in 15 day(s) (Fri 24 Jun 2022 14:48:24 GMT +0000). [14:52:38] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.50, 6.95, 6.57 [14:54:34] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.36, 5.87, 6.22 [15:05:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [15:08:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.83, 1.99 [15:17:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.51, 1.80, 1.99 [15:18:14] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.15, 1.74, 1.60 [15:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.13, 2.16, 2.03 [15:20:10] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.76, 1.39, 1.48 [15:23:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.36, 1.82, 1.91 [15:23:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.37, 9.78, 8.73 [15:24:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.63, 7.42, 6.25 [15:24:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.27, 2.13, 1.53 [15:25:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.94, 9.81, 8.87 [15:26:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.28, 7.03, 6.26 [15:26:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.88, 1.52 [15:28:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.58, 1.36, 1.36 [15:28:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.83, 1.96 [15:29:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 2.00, 2.00 [15:31:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.44, 2.12, 2.04 [15:31:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.64, 10.26, 9.37 [15:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.83, 1.92 [15:34:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.73, 6.62, 6.41 [15:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.81, 1.90 [15:34:26] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [15:34:26] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:35:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.52, 9.86, 9.43 [15:41:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.51, 1.79, 1.94 [15:44:04] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.93, 3.48, 3.11 [15:46:03] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.42, 3.82, 3.28 [15:47:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.34, 2.79, 2.24 [15:48:03] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.89, 3.50, 3.23 [15:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.79, 2.12, 1.94 [15:50:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.88, 6.66, 6.46 [15:52:01] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.41, 3.34, 3.26 [15:52:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.21, 6.68, 6.50 [15:54:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.95, 1.95 [15:57:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.85, 1.96 [15:59:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.27, 2.06, 2.03 [15:59:41] [02puppet] 07Universal-Omega opened pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659 [15:59:41] [url] Page not found · GitHub · GitHub | github.com [16:00:06] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.53, 1.73, 1.29 [16:00:33] [02puppet] 07Universal-Omega synchronize pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659 [16:00:33] [url] Page not found · GitHub · GitHub | github.com [16:01:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.89, 1.96 [16:01:07] [02puppet] 07Universal-Omega edited pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659 [16:01:08] [url] Page not found · GitHub · GitHub | github.com [16:02:01] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.97, 1.76, 1.35 [16:03:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.82, 2.29, 2.11 [16:03:57] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.28, 1.57, 1.33 [16:08:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.50, 1.69 [16:12:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.65, 1.72 [16:16:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.50, 1.61, 1.69 [16:20:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.76, 2.23, 1.93 [16:22:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.16, 6.78, 6.32 [16:22:18] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.59, 2.50, 1.85 [16:24:14] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.89, 1.96, 1.73 [16:26:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.25, 7.54, 6.71 [16:26:10] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.65, 1.65 [16:28:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.05, 7.40, 6.77 [16:30:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.14, 6.67, 6.58 [16:33:28] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [16:34:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.00, 7.14, 6.80 [16:34:52] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.96, 2.16, 1.87 [16:36:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.06, 6.53, 6.63 [16:36:48] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.22, 1.78, 1.75 [16:38:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.40, 3.27, 3.93 [16:38:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.93, 1.46, 1.64 [16:40:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 4.20, 4.19 [16:40:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.81, 1.56, 1.17 [16:42:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.21, 1.08 [16:43:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.46, 1.92 [16:43:05] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [16:43:36] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.m.miraheze.org' expires in 15 day(s) (Fri 24 Jun 2022 16:40:49 GMT +0000). [16:47:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:48:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.46, 3.33, 3.85 [16:49:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.23, 1.30, 1.70 [16:49:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.60, 1.99, 1.46 [16:51:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.64, 1.40 [16:55:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.42, 1.86, 1.58 [16:57:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.87, 2.02, 1.79 [16:57:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.69, 1.54 [16:59:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.73, 4.00 [16:59:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.33, 1.74, 1.72 [17:00:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 3.70, 3.68 [17:02:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [17:03:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.95, 1.57, 1.68 [17:04:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.74, 1.97 [17:05:31] [02puppet] 07JohnFLewis commented on pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659#issuecomment-1150174061 [17:05:31] [url] Move `mediawiki::jobqueue::chron` include to redis role by Universal-Omega · Pull Request #2659 · miraheze/puppet · GitHub | github.com [17:06:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.26, 3.52 [17:07:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.48, 3.73 [17:08:06] [02puppet] 07JohnFLewis commented on pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659#issuecomment-1150176515 [17:08:07] [url] Move `mediawiki::jobqueue::chron` include to redis role by Universal-Omega · Pull Request #2659 · miraheze/puppet · GitHub | github.com [17:09:36] [02puppet] 07JohnFLewis edited a comment on pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659#issuecomment-1150174061 [17:09:36] [url] Move `mediawiki::jobqueue::chron` include to redis role by Universal-Omega · Pull Request #2659 · miraheze/puppet · GitHub | github.com [17:10:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 3.74, 3.63 [17:11:04] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:11:20] [02puppet] 07Universal-Omega commented on pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659#issuecomment-1150179522 [17:11:21] [url] Move `mediawiki::jobqueue::chron` include to redis role by Universal-Omega · Pull Request #2659 · miraheze/puppet · GitHub | github.com [17:11:22] [02puppet] 07Universal-Omega closed pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659 [17:11:22] [url] Page not found · GitHub · GitHub | github.com [17:12:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.38, 3.44, 3.53 [17:12:15] [02puppet] 07Universal-Omega commented on pull request 03#2659: Move `mediawiki::jobqueue::chron` include to redis role - 13https://github.com/miraheze/puppet/pull/2659#issuecomment-1150180431 [17:12:15] [url] Move `mediawiki::jobqueue::chron` include to redis role by Universal-Omega · Pull Request #2659 · miraheze/puppet · GitHub | github.com [17:13:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.04, 1.81, 1.70 [17:14:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.87, 3.68 [17:15:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.79, 1.78, 1.70 [17:17:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.96, 1.48, 1.61 [17:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.97, 1.89 [17:22:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.78, 10.13, 8.92 [17:24:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 2.00, 1.95 [17:24:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.89, 9.58, 8.85 [17:27:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/055a77ba45e9 [17:27:09] [02miraheze/mw-config] 07Universal-Omega 03055a77b - Add 1.38 upgrade site notice [17:27:10] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [17:27:10] [url] Page not found · GitHub · GitHub | github.com [17:27:12] [02mw-config] 07Universal-Omega opened pull request 03#4740: Add 1.38 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/4740 [17:27:13] [url] Page not found · GitHub · GitHub | github.com [17:28:29] miraheze/mw-config - Universal-Omega the build passed. [17:29:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.40, 1.76, 1.69 [17:29:20] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.57, 3.40, 3.06 [17:30:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.65, 1.32 [17:30:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/055a77ba45e9...a9b53e82a66c [17:30:48] [url] Comparing 055a77ba45e9...a9b53e82a66c · miraheze/mw-config · GitHub | github.com [17:30:49] [02miraheze/mw-config] 07Universal-Omega 03a9b53e8 - Update Sitenotice.php [17:30:51] [02mw-config] 07Universal-Omega synchronize pull request 03#4740: Add 1.38 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/4740 [17:30:51] [url] Page not found · GitHub · GitHub | github.com [17:31:03] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.82, 7.00, 6.23 [17:32:02] miraheze/mw-config - Universal-Omega the build passed. [17:32:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.52, 2.34, 1.59 [17:32:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.79, 1.85 [17:33:19] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.35, 3.75, 3.25 [17:34:09] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.51, 1.93, 1.53 [17:34:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.79, 1.85 [17:35:17] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.13, 3.48, 3.22 [17:36:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.29, 2.40, 1.74 [17:37:16] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.18, 3.34, 3.19 [17:38:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.78, 1.58 [17:38:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.25, 6.79, 6.49 [17:39:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.45, 1.60, 1.67 [17:40:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.57, 1.53 [17:41:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:44:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.80, 1.78 [17:47:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.88, 1.67, 1.66 [17:47:11] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.38, 3.63, 3.33 [17:48:12] [02puppet] 07Universal-Omega opened pull request 03#2660: mwupgradetool: use canary not servers=skip - 13https://github.com/miraheze/puppet/pull/2660 [17:48:13] [url] Page not found · GitHub · GitHub | github.com [17:48:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.78, 7.38, 6.75 [17:51:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.32, 1.82, 1.71 [17:53:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.85, 1.74 [17:55:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.04, 1.98, 1.80 [17:55:07] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.98, 3.91, 3.74 [17:56:21] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.20, 7.94, 7.17 [17:57:09] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.84, 1.78 [17:58:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.90, 1.99 [17:58:17] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.93, 7.62, 7.15 [17:59:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.83, 1.77 [18:01:07] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.73, 3.68, 3.61 [18:01:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [18:02:10] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.44, 6.46, 6.77 [18:04:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 1.98, 1.97 [18:05:05] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.76, 3.70, 3.64 [18:09:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.59, 1.93, 1.93 [18:11:02] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.20, 2.94, 3.32 [18:11:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.87, 2.38, 2.09 [18:15:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.19, 1.69, 1.87 [18:21:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.10, 1.61, 1.76 [18:22:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.24, 6.51, 6.40 [18:23:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.78, 1.73, 1.79 [18:24:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.99, 6.58, 6.46 [18:25:04] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.50, 1.88, 1.83 [18:28:07] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9984b91ae0cb...79a6b4d729a8 [18:28:08] [url] Comparing 9984b91ae0cb...79a6b4d729a8 · miraheze/ImportDump · GitHub | github.com [18:28:09] [02miraheze/ImportDump] 07Universal-Omega 0379a6b4d - Utilise `Html::noticeBox()` [18:29:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.26, 1.67, 1.77 [18:31:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.50, 1.69 [18:31:52] miraheze/ImportDump - Universal-Omega the build has errored. [18:32:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.46, 6.96, 6.65 [18:34:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.39, 6.73, 6.60 [18:35:44] PROBLEM - wiki.horizonsend.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.horizonsend.net could not be found [18:36:59] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/79a6b4d729a8...0c6333750e28 [18:36:59] [url] Comparing 79a6b4d729a8...0c6333750e28 · miraheze/ImportDump · GitHub | github.com [18:37:00] [02miraheze/ImportDump] 07Universal-Omega 030c63337 - Fix [18:38:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.84, 8.13, 7.17 [18:40:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.98, 7.26, 6.97 [18:44:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.01, 6.19, 6.62 [18:47:12] miraheze/ImportDump - Universal-Omega the build passed. [18:50:12] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.74, 1.41, 0.99 [18:52:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.73, 1.12, 0.93 [18:52:55] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:53:26] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/2c925e635f9b [18:53:27] [02miraheze/ImportDump] 07Universal-Omega 032c925e6 - Update ImportDumpRequestQueuePager.php [18:53:29] [02ImportDump] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ImportDump [18:53:29] [url] Page not found · GitHub · GitHub | github.com [18:54:53] [02ImportDump] 07Universal-Omega opened pull request 03#4: Replace usage of `getConnectionRef()` with `getConnection()` - 13https://github.com/miraheze/ImportDump/pull/4 [18:54:53] ... [18:55:40] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2c925e635f9b...4ebdaa55d785 [18:55:41] [url] Comparing 2c925e635f9b...4ebdaa55d785 · miraheze/ImportDump · GitHub | github.com [18:55:42] [02miraheze/ImportDump] 07Universal-Omega 034ebdaa5 - Update ImportDumpRequestManager.php [18:55:43] [02ImportDump] 07Universal-Omega synchronize pull request 03#4: Replace usage of `getConnectionRef()` with `getConnection()` - 13https://github.com/miraheze/ImportDump/pull/4 [18:55:44] [url] Page not found · GitHub · GitHub | github.com [18:56:14] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4ebdaa55d785...57a8c8ad5232 [18:56:14] [url] Comparing 4ebdaa55d785...57a8c8ad5232 · miraheze/ImportDump · GitHub | github.com [18:56:16] [02miraheze/ImportDump] 07Universal-Omega 0357a8c8a - Update SpecialRequestImportDump.php [18:56:17] [02ImportDump] 07Universal-Omega synchronize pull request 03#4: Replace usage of `getConnectionRef()` with `getConnection()` - 13https://github.com/miraheze/ImportDump/pull/4 [18:56:17] [url] Page not found · GitHub · GitHub | github.com [18:57:01] [02ImportDump] 07Universal-Omega edited pull request 03#4: Replace usage of `getConnectionRef()` with `getConnection()` - 13https://github.com/miraheze/ImportDump/pull/4 [18:57:02] [url] Page not found · GitHub · GitHub | github.com [19:02:55] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.28, 6.84, 6.44 [19:04:55] miraheze/ImportDump - Universal-Omega the build passed. [19:05:45] [02ImportDump] 07Universal-Omega closed pull request 03#4: Replace usage of `getConnectionRef()` with `getConnection()` - 13https://github.com/miraheze/ImportDump/pull/4 [19:05:45] [url] Page not found · GitHub · GitHub | github.com [19:05:47] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ImportDump/compare/0c6333750e28...a3c10dd92fe6 [19:05:47] [url] Comparing 0c6333750e28...a3c10dd92fe6 · miraheze/ImportDump · GitHub | github.com [19:05:48] [02miraheze/ImportDump] 07Universal-Omega 03a3c10dd - Replace usage of `getConnectionRef()` with `getConnection()` (#4) [19:05:50] [02ImportDump] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ImportDump [19:05:50] [url] Page not found · GitHub · GitHub | github.com [19:05:51] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:06:11] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/26b92a59d693 [19:06:13] [02miraheze/ImportDump] 07Universal-Omega 0326b92a5 - Enable remaining phan checks [19:06:14] [02ImportDump] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ImportDump [19:06:14] ... [19:06:16] [02ImportDump] 07Universal-Omega opened pull request 03#5: Enable remaining phan checks - 13https://github.com/miraheze/ImportDump/pull/5 [19:06:16] ... [19:06:48] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.00, 6.45, 6.42 [19:09:29] miraheze/ImportDump - Universal-Omega the build has errored. [19:09:48] miraheze/ImportDump - Universal-Omega the build has errored. [19:10:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.83, 1.76, 1.65 [19:10:32] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.21, 3.64, 3.05 [19:11:43] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/fab955e85bac [19:11:45] [02miraheze/ImportDump] 07Universal-Omega 03fab955e - Fix interwiki prefix map check [19:11:46] [02ImportDump] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/ImportDump [19:11:46] [url] Page not found · GitHub · GitHub | github.com [19:11:48] [02ImportDump] 07Universal-Omega opened pull request 03#6: Fix interwiki prefix map check - 13https://github.com/miraheze/ImportDump/pull/6 [19:11:48] ... [19:12:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.32, 1.58, 1.60 [19:12:31] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.75, 3.85, 3.21 [19:15:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.56, 3.92 [19:15:12] miraheze/ImportDump - Universal-Omega the build has errored. [19:16:31] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.29, 7.29, 6.77 [19:18:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.86, 6.99, 6.71 [19:20:24] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.96, 6.52, 6.57 [19:20:55] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [19:22:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4740: Add 1.38 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/4740 [19:22:17] [url] Page not found · GitHub · GitHub | github.com [19:22:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a9b53e82a66c...485c8fa9fd38 [19:22:18] [url] Comparing a9b53e82a66c...485c8fa9fd38 · miraheze/mw-config · GitHub | github.com [19:22:20] [02miraheze/mw-config] 07Universal-Omega 03485c8fa - Update Sitenotice.php [19:22:26] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.51, 3.17, 3.17 [19:23:26] miraheze/mw-config - Universal-Omega the build passed. [19:24:34] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 5.23, 3.71, 2.73 [19:27:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.14, 4.84, 4.17 [19:28:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.62, 3.88, 3.05 [19:28:37] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.82, 1.82, 1.70 [19:30:35] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.56, 1.99, 1.77 [19:32:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.19, 6.92, 6.68 [19:32:32] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 2.00, 1.95, 1.78 [19:34:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.50, 3.97 [19:36:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.68, 3.96 [19:36:09] miraheze/ImportDump - Universal-Omega the build has errored. [19:37:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.25, 9.90, 8.91 [19:38:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.46, 3.86 [19:38:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.24, 7.33, 6.93 [19:38:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.20, 2.26, 1.94 [19:39:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.80, 9.51, 8.89 [19:42:34] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.67, 2.82, 3.04 [19:44:09] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.45, 7.30, 7.18 [19:50:37] [02mw-config] 07Universal-Omega closed pull request 03#4740: Add 1.38 upgrade site notice - 13https://github.com/miraheze/mw-config/pull/4740 [19:50:37] [url] Page not found · GitHub · GitHub | github.com [19:50:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7d153127d7ae...6a9851e0945f [19:50:39] [url] Comparing 7d153127d7ae...6a9851e0945f · miraheze/mw-config · GitHub | github.com [19:50:40] [02miraheze/mw-config] 07Universal-Omega 036a9851e - Add 1.38 upgrade site notice (#4740) [19:50:41] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [19:50:43] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [19:50:43] [url] Page not found · GitHub · GitHub | github.com [19:51:44] miraheze/mw-config - Universal-Omega the build passed. [19:52:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.23, 4.28, 3.98 [19:52:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.23, 6.21, 6.72 [19:52:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.80, 10.52, 9.82 [19:52:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 9.69, 9.19 [19:53:07] !log [@mwtask111] starting deploy of {'config': True} to all [19:53:43] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 36s [19:53:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.73, 3.81 [19:55:49] !log [@test101] starting deploy of {'config': True} to all [19:55:52] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 3s [19:55:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:56:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.99, 9.24, 9.51 [19:56:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.65, 10.23, 9.45 [19:57:37] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [19:58:01] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.12, 1.62, 1.90 [19:58:09] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.26, 7.75, 7.16 [19:58:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.81, 10.09, 9.49 [20:00:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.25, 7.49, 7.15 [20:00:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.13, 10.14, 9.66 [20:02:55] PROBLEM - wiki.zergmafia.top - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zergmafia.top' expires in 15 day(s) (Fri 24 Jun 2022 19:58:11 GMT +0000). [20:03:03] PROBLEM - wiki.pixlies.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pixlies.net' expires in 15 day(s) (Fri 24 Jun 2022 19:33:48 GMT +0000). [20:03:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.29, 1.79, 1.84 [20:05:51] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.83, 1.86 [20:06:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.34, 6.46, 6.76 [20:07:38] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:08:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.64, 3.69 [20:09:47] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.74, 1.18, 1.58 [20:10:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.99, 3.85 [20:12:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.65, 4.06, 3.90 [20:14:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.96, 3.89 [20:14:35] PROBLEM - wikifencing.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikifencing.ru' expires in 15 day(s) (Fri 24 Jun 2022 20:04:19 GMT +0000). [20:17:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.42, 3.95 [20:18:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.29, 1.87, 1.68 [20:20:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.35, 2.67, 3.30 [20:20:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.56, 1.59 [20:21:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6a9851e0945f...a68d0d0ebd26 [20:21:20] [url] Comparing 6a9851e0945f...a68d0d0ebd26 · miraheze/mw-config · GitHub | github.com [20:21:21] [02miraheze/mw-config] 07Universal-Omega 03a68d0d0 - Update SemanticMediaWiki.php [20:21:44] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [20:21:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:52] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [20:21:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:29] miraheze/mw-config - Universal-Omega the build passed. [20:22:39] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [20:22:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.85, 3.91 [20:24:12] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=0) [20:24:13] !log [universalomega@mwtask111] starting deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all [20:24:26] !log [universalomega@mwtask111] finished deploy of {'files': '../mediawiki/w/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 12s [20:24:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:25:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.61, 3.82 [20:25:28] !log [@test101] starting deploy of {'config': True} to all [20:25:29] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [20:25:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.58, 6.21, 5.71 [20:27:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.55, 9.25, 8.37 [20:28:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.62, 6.05, 5.72 [20:28:53] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.79, 3.15, 2.74 [20:29:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.85, 9.38, 8.52 [20:30:52] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.35, 2.79, 2.65 [20:30:56] PROBLEM - nl.xliving.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nl.xliving.org' expires in 15 day(s) (Fri 24 Jun 2022 20:27:17 GMT +0000). [20:31:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.67, 3.96 [20:32:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9ce46bd80c7a [20:32:29] [02miraheze/mw-config] 07Universal-Omega 039ce46bd - Update skin settings for MediaWiki 1.38 [20:32:30] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [20:32:31] [url] Page not found · GitHub · GitHub | github.com [20:32:32] [02mw-config] 07Universal-Omega opened pull request 03#4741: Update skin settings for MediaWiki 1.38 - 13https://github.com/miraheze/mw-config/pull/4741 [20:32:32] [url] Page not found · GitHub · GitHub | github.com [20:33:42] miraheze/mw-config - Universal-Omega the build passed. [20:33:45] RECOVERY - wiki.horizonsend.net - reverse DNS on sslhost is OK: SSL OK - wiki.horizonsend.net reverse DNS resolves to cp20.miraheze.org - CNAME OK [20:35:22] PROBLEM - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pbptanarchy.tk' expires in 15 day(s) (Fri 24 Jun 2022 20:14:25 GMT +0000). [20:37:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.36, 3.55 [20:39:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.19, 3.46 [20:39:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 4.06, 3.95 [20:40:21] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.01, 10.21, 9.10 [20:41:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.24, 2.93, 3.34 [20:41:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.85, 3.89 [20:42:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.38, 9.83, 9.12 [20:43:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.05, 4.37, 4.06 [20:45:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.43, 3.45 [20:45:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9ce46bd80c7a...989c4e569a92 [20:45:56] [url] Comparing 9ce46bd80c7a...989c4e569a92 · miraheze/mw-config · GitHub | github.com [20:45:57] [02miraheze/mw-config] 07Universal-Omega 03989c4e5 - Update LocalSettings.php [20:45:59] [02mw-config] 07Universal-Omega synchronize pull request 03#4741: Update skin settings for MediaWiki 1.38 - 13https://github.com/miraheze/mw-config/pull/4741 [20:45:59] [url] Page not found · GitHub · GitHub | github.com [20:46:08] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.09, 7.16, 6.32 [20:47:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.08, 3.15, 3.34 [20:47:07] miraheze/mw-config - Universal-Omega the build passed. [20:47:20] !log [universalomega@test101] starting deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-2', 'config': True} to all [20:47:23] !log [universalomega@test101] finished deploy of {'pull': 'config', 'branch': 'Universal-Omega-patch-2', 'config': True} to all - SUCCESS in 3s [20:47:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.75, 3.92 [20:47:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.60, 11.38, 10.00 [20:48:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.22, 7.11, 6.40 [20:49:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.30, 5.39, 4.49 [20:50:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.04, 6.63, 6.31 [20:50:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.29, 4.69, 3.79 [20:51:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 4.14, 3.72 [20:51:41] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 5.12, 3.76, 3.06 [20:51:45] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.38, 10.68, 9.49 [20:52:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.77, 9.99, 9.74 [20:53:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.65, 10.20, 9.45 [20:54:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.36, 5.32, 2.66 [20:54:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.99, 3.69 [20:55:41] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.85, 3.55, 3.16 [20:56:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.48, 4.51, 3.91 [20:56:35] !log [@test101] starting deploy of {'config': True} to all [20:56:36] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [20:56:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:59:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4741: Update skin settings for MediaWiki 1.38 - 13https://github.com/miraheze/mw-config/pull/4741 [20:59:23] [url] Page not found · GitHub · GitHub | github.com [20:59:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/989c4e569a92...d10ca5e4d6d2 [20:59:25] [url] Comparing 989c4e569a92...d10ca5e4d6d2 · miraheze/mw-config · GitHub | github.com [20:59:26] [02miraheze/mw-config] 07Universal-Omega 03d10ca5e - Update ManageWikiSettings.php [20:59:41] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.45, 3.80, 3.33 [21:00:35] miraheze/mw-config - Universal-Omega the build passed. [21:00:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4741: Update skin settings for MediaWiki 1.38 - 13https://github.com/miraheze/mw-config/pull/4741 [21:00:39] ... [21:00:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d10ca5e4d6d2...7779e53bdc97 [21:00:41] [url] Comparing d10ca5e4d6d2...7779e53bdc97 · miraheze/mw-config · GitHub | github.com [21:00:42] [02miraheze/mw-config] 07Universal-Omega 037779e53 - Update LocalSettings.php [21:01:32] PROBLEM - encyclopediarobotica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediarobotica.org' expires in 15 day(s) (Fri 24 Jun 2022 20:32:37 GMT +0000). [21:01:41] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 2.65, 3.45, 3.27 [21:01:49] miraheze/mw-config - Universal-Omega the build passed. [21:02:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.06, 1.85, 1.98 [21:03:41] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.83, 3.25, 3.21 [21:09:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.71, 3.96 [21:10:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.01, 1.53, 1.71 [21:10:36] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.89, 6.87, 6.55 [21:11:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.76, 3.94 [21:12:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.69, 1.19, 1.56 [21:12:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.02, 9.30, 8.08 [21:13:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.87, 3.96 [21:14:29] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.33, 7.27, 6.77 [21:14:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.68, 9.02, 8.12 [21:15:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 4.38, 4.16 [21:17:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:19:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.82, 4.00 [21:22:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.31, 3.82 [21:24:40] !log [@test101] starting deploy of {'config': True} to all [21:24:41] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [21:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:08] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.38, 7.11, 7.34 [21:27:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [21:29:09] !log [universalomega@test101] starting deploy of {'config': True} to all [21:29:10] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [21:29:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:56] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.96, 1.57 [21:32:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 2.78, 3.34 [21:33:53] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.01, 1.54, 1.46 [21:34:08] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.50, 5.93, 6.68 [21:35:55] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [21:35:55] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:36:35] [02miraheze/MirahezeMagic] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/setup-python-4 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/c6e0148905bb [21:36:36] [02miraheze/MirahezeMagic] 07dependabot[bot] 03c6e0148 - Bump actions/setup-python from 3 to 4 [21:36:38] [02MirahezeMagic] 07dependabot[bot] opened pull request 03#339: Bump actions/setup-python from 3 to 4 - 13https://github.com/miraheze/MirahezeMagic/pull/339 [21:36:38] [url] Page not found · GitHub · GitHub | github.com [21:36:40] [02MirahezeMagic] 07dependabot[bot] created branch 03dependabot/github_actions/actions/setup-python-4 - 13https://github.com/miraheze/MirahezeMagic [21:36:40] [url] Page not found · GitHub · GitHub | github.com [21:36:41] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#339: Bump actions/setup-python from 3 to 4 - 13https://github.com/miraheze/MirahezeMagic/pull/339 [21:36:41] [url] Page not found · GitHub · GitHub | github.com [21:36:42] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#339: Bump actions/setup-python from 3 to 4 - 13https://github.com/miraheze/MirahezeMagic/pull/339 [21:36:43] ... [21:38:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 4.04, 3.72 [21:39:58] miraheze/MirahezeMagic - dependabot[bot] the build has errored. [21:42:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.95, 3.79, 3.72 [21:42:14] CosmicAlpha: magic CI fails [21:43:26] [02puppet] 07RhinosF1 opened pull request 03#2661: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/2661 [21:43:27] [url] Page not found · GitHub · GitHub | github.com [21:43:39] paladox: ^ should be fine [21:43:52] 2660 can go too [21:44:49] Not just magic, it is failing other places also due to something upstream I think. (`Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Access denied for user 'wikiuser'@'localhost' to database 'centralauth' (localhost:/tmp/quibble-mysql-jgcyywmh/socket)`) [21:45:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.30, 4.41, 3.83 [21:45:18] [02puppet] 07paladox closed pull request 03#2661: Update workflow.yml - 13https://github.com/miraheze/puppet/pull/2661 [21:45:18] [url] Page not found · GitHub · GitHub | github.com [21:45:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bd0dca091ee3...0b4dda583244 [21:45:20] [url] Comparing bd0dca091ee3...0b4dda583244 · miraheze/puppet · GitHub | github.com [21:45:21] [02miraheze/puppet] 07RhinosF1 030b4dda5 - Update workflow.yml (#2661) [21:46:38] paladox: do you mind also merging https://github.com/miraheze/puppet/pull/2660? [21:46:38] [url] mwupgradetool: use canary not servers=skip by Universal-Omega · Pull Request #2660 · miraheze/puppet · GitHub | github.com [21:46:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0b4dda583244...cbe91142b516 [21:46:56] [url] Comparing 0b4dda583244...cbe91142b516 · miraheze/puppet · GitHub | github.com [21:46:57] [02miraheze/puppet] 07Universal-Omega 03cbe9114 - mwupgradetool: use canary not servers=skip (#2660) [21:46:59] [02puppet] 07paladox closed pull request 03#2660: mwupgradetool: use canary not servers=skip - 13https://github.com/miraheze/puppet/pull/2660 [21:46:59] [url] Page not found · GitHub · GitHub | github.com [21:47:02] Thanks! [21:47:23] [02puppet] 07RhinosF1 opened pull request 03#2662: mw-sre: support py 3.11 - 13https://github.com/miraheze/puppet/pull/2662 [21:47:24] [url] Page not found · GitHub · GitHub | github.com [21:47:51] [02puppet] 07RhinosF1 synchronize pull request 03#2662: mw-sre: support py 3.11 - 13https://github.com/miraheze/puppet/pull/2662 [21:47:51] [url] Page not found · GitHub · GitHub | github.com [21:48:16] Ty paladox [21:49:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.74, 3.68 [21:51:21] [02puppet] 07RhinosF1 synchronize pull request 03#2662: mw-sre: support py 3.11 - 13https://github.com/miraheze/puppet/pull/2662 [21:51:21] [url] Page not found · GitHub · GitHub | github.com [21:53:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.65, 3.63 [21:54:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.69, 3.68 [21:54:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.45, 1.19 [21:56:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.47, 3.58 [21:56:08] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.13, 1.95, 1.39 [21:57:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.46, 3.58 [21:58:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.55, 1.31 [21:58:40] [02puppet] 07RhinosF1 synchronize pull request 03#2662: mw-sre: support py 3.11 - 13https://github.com/miraheze/puppet/pull/2662 [21:58:40] [url] Page not found · GitHub · GitHub | github.com [21:58:57] [02puppet] 07RhinosF1 commented on pull request 03#2662: mw-sre: support py 3.11 - 13https://github.com/miraheze/puppet/pull/2662#issuecomment-1150457302 [21:58:58] [url] mw-sre: support py 3.11 by RhinosF1 · Pull Request #2662 · miraheze/puppet · GitHub | github.com [22:00:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.81, 3.66 [22:00:12] CosmicAlpha: we've got like a year to fix ^ [22:00:35] 3.11 will likely end up being the default in bookworm / Debian 11 [22:01:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 4.02, 3.76 [22:01:18] I've made it error on warnings so we can fix them and don't loose that requests has a broken dependancy [22:02:41] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±9] 13https://github.com/miraheze/ssl/compare/bfa4a03260a9...ed58eeb37b5f [22:02:41] [url] Comparing bfa4a03260a9...ed58eeb37b5f · miraheze/ssl · GitHub | github.com [22:02:43] [02miraheze/ssl] 07Universal-Omega 03ed58eeb - Renew certificates [22:02:58] RhinosF1: OK. [22:04:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:05:08] CosmicAlpha: can you make magic CI work [22:05:50] RhinosF1: I just said it was probably upstream. It worked yesterday, no changes since, and it is related to CentralAuth database access it says. [22:06:50] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed58eeb37b5f...c596bd910c60 [22:06:51] [url] Comparing ed58eeb37b5f...c596bd910c60 · miraheze/ssl · GitHub | github.com [22:06:52] [02miraheze/ssl] 07Universal-Omega 03c596bd9 - Renew housing.wiki certificate [22:07:50] CosmicAlpha: ah [22:07:53] Missed that [22:08:02] I guess bug needs raising [22:12:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.80, 3.92 [22:12:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.30, 1.56, 1.92 [22:14:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 4.04, 4.00 [22:14:55] [02dns] 07Universal-Omega opened pull request 03#287: add acme challenges for m.miraheze.org - 13https://github.com/miraheze/dns/pull/287 [22:14:55] [url] Page not found · GitHub · GitHub | github.com [22:15:31] paladox: ^ can you please merge and deploy if you are still around? [22:15:47] [02dns] 07paladox closed pull request 03#287: add acme challenges for m.miraheze.org - 13https://github.com/miraheze/dns/pull/287 [22:15:47] ... [22:15:48] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/02f49ab7c189...0b2be63ae0b0 [22:15:49] [url] Comparing 02f49ab7c189...0b2be63ae0b0 · miraheze/dns · GitHub | github.com [22:15:50] [02miraheze/dns] 07Universal-Omega 030b2be63 - add acme challenges for m.miraheze.org (#287) [22:17:05] CosmicAlpha: done [22:18:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.96, 3.93, 3.95 [22:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.85, 1.90 [22:19:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.62, 3.95 [22:20:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 4.09, 3.99 [22:22:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.99, 3.30, 3.71 [22:22:11] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c596bd910c60...68662d9ad456 [22:22:11] [url] Comparing c596bd910c60...68662d9ad456 · miraheze/ssl · GitHub | github.com [22:22:12] [02miraheze/ssl] 07Universal-Omega 0368662d9 - Renew *.m.miraheze.org [22:22:20] Thanks paladox [22:24:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.99, 1.98 [22:25:42] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk [22:27:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.00, 2.96, 3.84 [22:28:56] RECOVERY - nl.xliving.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nl.xliving.org' will expire on Tue 06 Sep 2022 20:54:18 GMT +0000. [22:30:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.27, 3.49 [22:30:56] RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zergmafia.top' will expire on Tue 06 Sep 2022 20:57:49 GMT +0000. [22:31:03] RECOVERY - wiki.pixlies.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pixlies.net' will expire on Tue 06 Sep 2022 20:56:25 GMT +0000. [22:31:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.58, 2.76, 3.29 [22:31:20] RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Tue 06 Sep 2022 20:51:36 GMT +0000. [22:31:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.84, 3.37, 3.78 [22:32:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.22, 3.44 [22:33:14] RECOVERY - housing.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'housing.wiki' will expire on Tue 06 Sep 2022 21:04:59 GMT +0000. [22:33:21] RECOVERY - wiki.pbptanarchy.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pbptanarchy.tk' will expire on Tue 06 Sep 2022 20:52:59 GMT +0000. [22:33:54] RECOVERY - wiki.wisrail.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wisrail.de' will expire on Tue 06 Sep 2022 21:01:07 GMT +0000. [22:34:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.41, 3.47 [22:35:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.70, 3.81 [22:35:44] PROBLEM - wiki.horizonsend.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.horizonsend.net could not be found [22:36:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.35, 3.44 [22:38:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.16, 3.07, 3.34 [22:42:35] RECOVERY - wikifencing.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wikifencing.ru' will expire on Tue 06 Sep 2022 20:55:14 GMT +0000. [22:43:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.95, 3.44, 3.56 [22:47:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.25, 3.50 [22:48:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.62, 1.98, 1.85 [22:49:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.67, 4.83, 4.01 [22:50:14] RECOVERY - wiki.lu9.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lu9.io' will expire on Tue 06 Sep 2022 20:59:54 GMT +0000. [22:50:27] RECOVERY - wildbowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wildbowiki.net' will expire on Tue 06 Sep 2022 20:59:02 GMT +0000. [22:51:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.78, 3.35 [22:52:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.51, 3.35 [22:53:44] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:54:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.91, 1.90 [22:54:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.27, 3.29 [22:55:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.62, 3.42 [22:55:46] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:56:32] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.08, 7.35, 6.46 [22:57:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 4.18, 3.64 [22:58:21] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:58:28] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.89, 7.15, 6.50 [22:58:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.75, 3.68, 3.45 [22:58:49] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.81, 1.49, 1.20 [22:59:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.71, 3.54 [22:59:44] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:00:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.76, 2.96, 3.21 [23:00:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.75, 1.20, 1.12 [23:03:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 2.92, 3.24 [23:04:17] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.87, 6.76, 6.57 [23:04:59] RECOVERY - wiki.horizonsend.net - reverse DNS on sslhost is OK: SSL OK - wiki.horizonsend.net reverse DNS resolves to cp21.miraheze.org - CNAME OK [23:06:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.44, 3.36 [23:07:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.44, 3.38 [23:08:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.02, 3.33, 3.34 [23:11:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 3.09, 3.27 [23:11:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.75, 3.71, 3.95 [23:13:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 4.14, 4.08 [23:14:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.56, 1.70 [23:15:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.91, 1.56 [23:17:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.33, 1.62, 1.49 [23:18:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.71, 2.04, 1.86 [23:21:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.66, 3.94 [23:21:43] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.59, 4.05, 4.03 [23:23:44] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:25:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.78, 3.93 [23:25:42] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk [23:26:19] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:27:43] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:29:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.45, 3.72 [23:32:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.41, 3.11 [23:34:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 3.20, 3.07 [23:36:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.90, 2.00 [23:38:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 3.91, 3.35 [23:39:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.67, 3.35 [23:40:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.51, 3.27 [23:41:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.15, 3.08, 3.17 [23:42:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 4.12, 3.51 [23:44:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.83, 3.47 [23:46:08] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 3.32, 3.33 [23:46:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.23, 2.01, 2.00 [23:52:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.55, 3.37 [23:53:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.00, 3.89, 3.41 [23:54:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 3.78, 3.46