[00:03:51] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 342 failures. Last run 3 minutes ago with 342 failures. Failed resources (up to 3 shown): File[wiki.thedev.gq_private],File[thestarsareright.org],File[thestarsareright.org_private],File[electrowiki.org] [00:05:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.23, 2.64 [00:06:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:07:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 2.96, 2.61 [00:09:27] [discord] Can someone check if it's possible to unban RcGcDb? I would like to restart it for all the other wikis using it [00:09:27] [discord] https://phabricator.miraheze.org/T8851#179059 [00:09:28] [url] ⚓ T8851 db111 insanely high load | phabricator.miraheze.org [00:11:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.78, 1.46 [00:12:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.65, 8.30, 7.99 [00:13:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.68, 1.46 [00:14:15] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.62, 9.44, 8.36 [00:14:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.58, 8.31, 8.05 [00:15:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.44, 3.08 [00:16:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.58, 8.72, 8.24 [00:17:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 3.27, 3.07 [00:17:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.55, 3.12 [00:18:11] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.57, 8.31, 8.14 [00:18:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/548a44da856c [00:18:58] [02miraheze/puppet] 07paladox 03548a44d - varnish: Unban wiki-bot [00:19:00] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [00:19:01] [02puppet] 07paladox opened pull request 03#2443: varnish: Unban wiki-bot - 13https://github.com/miraheze/puppet/pull/2443 [00:19:09] [02puppet] 07paladox closed pull request 03#2443: varnish: Unban wiki-bot - 13https://github.com/miraheze/puppet/pull/2443 [00:19:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5ea183da88f5...cb0ddffbc7cd [00:19:12] [02miraheze/puppet] 07paladox 03cb0ddff - varnish: Unban wiki-bot (#2443) [00:19:14] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [00:19:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [00:19:20] @MarkusRost ^ [00:19:27] it'll deploy within the next hour [00:19:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.11, 3.38, 3.10 [00:19:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.54, 1.32 [00:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.41, 1.30 [00:22:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.64, 8.58, 8.10 [00:25:21] [discord] ok, thanks [00:26:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.36, 8.08, 8.02 [00:26:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:27:20] [02CreateWiki] 07Universal-Omega closed pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [00:27:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/compare/35542b34cf30...45072866216a [00:27:23] [02miraheze/CreateWiki] 07Universal-Omega 034507286 - Add RemoteWiki phpunit tests (#304) [00:27:25] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:27:26] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [00:30:23] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/7b523b36ed4e [00:30:25] [02miraheze/CreateWiki] 07Universal-Omega 037b523b3 - Add coverage report to CI [00:30:26] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [00:30:28] [02CreateWiki] 07Universal-Omega opened pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:31:42] miraheze/CreateWiki - Universal-Omega the build has errored. [00:31:48] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:32:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7b523b36ed4e...68dd442d716f [00:32:26] [02miraheze/CreateWiki] 07Universal-Omega 0368dd442 - Update mediawiki-tests.yml [00:32:28] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:35:15] miraheze/CreateWiki - Universal-Omega the build has errored. [00:35:31] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/68dd442d716f...90f4508dd837 [00:35:32] [02miraheze/CreateWiki] 07Universal-Omega 0390f4508 - Update mediawiki-tests.yml [00:35:34] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:37:36] miraheze/CreateWiki - Universal-Omega the build passed. [00:38:43] miraheze/CreateWiki - Universal-Omega the build has errored. [00:40:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.64, 8.79, 7.78 [00:41:08] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/90f4508dd837...f7038a275747 [00:41:09] [02miraheze/CreateWiki] 07Universal-Omega 03f7038a2 - Update mediawiki-tests.yml [00:41:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:41:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 8.21, 7.70 [00:42:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 8.57, 7.82 [00:43:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.17, 7.71, 7.57 [00:44:37] miraheze/CreateWiki - Universal-Omega the build has errored. [00:44:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.59, 10.20, 8.49 [00:47:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:48:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.03, 9.36, 8.58 [00:49:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.73, 10.19, 8.66 [00:50:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.99, 8.46, 8.34 [00:51:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.37, 9.20, 8.40 [00:53:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 9.37, 8.54 [00:54:36] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f7038a275747...978b248dea7e [00:54:37] [02miraheze/CreateWiki] 07Universal-Omega 03978b248 - Update mediawiki-tests.yml [00:54:39] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:55:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 8.47, 8.31 [00:56:36] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [00:56:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406069993972778 seconds [00:57:06] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/978b248dea7e...bf952689f109 [00:57:08] [02miraheze/CreateWiki] 07Universal-Omega 03bf95268 - Update mediawiki-tests.yml [00:57:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.45, 9.71, 9.13 [00:57:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:57:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/bf952689f109...ed8a63686e9a [00:57:51] [02miraheze/CreateWiki] 07Universal-Omega 03ed8a636 - Update mediawiki-tests.yml [00:57:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [00:57:59] miraheze/CreateWiki - Universal-Omega the build has errored. [01:01:17] [02CreateWiki] 07Universal-Omega closed pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [01:01:19] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [01:01:21] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:04:44] [02miraheze/CreateWiki] 07Universal-Omega pushed 037 commits to 03Universal-Omega-patch-2 [+0/-0/±7] 13https://github.com/miraheze/CreateWiki/compare/7b523b36ed4e^...ed8a63686e9a [01:04:45] [02miraheze/CreateWiki] 07Universal-Omega 037b523b3 - Add coverage report to CI [01:04:47] [02miraheze/CreateWiki] 07Universal-Omega 0368dd442 - Update mediawiki-tests.yml [01:04:48] [02miraheze/CreateWiki] 07Universal-Omega 0390f4508 - Update mediawiki-tests.yml [01:04:50] [02miraheze/CreateWiki] ... and 4 more commits. [01:04:51] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [01:04:53] [02CreateWiki] 07Universal-Omega reopened pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [01:05:27] miraheze/CreateWiki - Universal-Omega the build has errored. [01:07:41] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ed8a63686e9a...3063152a201e [01:07:43] [02miraheze/CreateWiki] 07Universal-Omega 033063152 - Update mediawiki-tests.yml [01:07:44] [02CreateWiki] 07Universal-Omega synchronize pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [01:08:15] miraheze/CreateWiki - Universal-Omega the build has errored. [01:11:01] miraheze/CreateWiki - Universal-Omega the build has errored. [01:12:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.23, 9.54, 8.35 [01:12:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.68, 9.83, 9.12 [01:13:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.91, 1.69, 1.20 [01:14:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.77, 9.65, 8.53 [01:14:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 9.24, 9.00 [01:15:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.57, 1.21 [01:15:10] [discord] I have restarted RcGcDb now [01:15:56] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 9.02, 8.31 [01:16:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.50, 8.69, 8.26 [01:16:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.87, 3.04 [01:17:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 9.19, 8.49 [01:17:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.46, 7.64, 7.94 [01:18:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.46, 2.99 [01:19:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.68, 1.40 [01:19:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.05, 7.68, 8.06 [01:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.26, 1.54, 1.39 [01:22:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.54, 9.75, 9.36 [01:22:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 2.98, 2.90 [01:23:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.16, 9.47, 8.78 [01:23:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.30, 8.69, 8.44 [01:25:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.17, 9.22, 8.80 [01:25:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.05, 9.59, 8.80 [01:26:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.55, 9.39, 9.35 [01:26:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [01:27:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.11, 9.36, 8.81 [01:28:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.78, 8.32, 8.14 [01:29:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.20, 9.88, 9.06 [01:31:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 9.60, 9.04 [01:31:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.08, 8.89, 8.74 [01:32:26] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.17, 8.40, 8.24 [01:36:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.33, 9.23 [01:37:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 10.00, 9.39 [01:37:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:38:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 9.07, 9.13 [01:38:24] [02CreateWiki] 07Universal-Omega closed pull request 03#307: Add coverage report to CI - 13https://github.com/miraheze/CreateWiki/pull/307 [01:38:27] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:38:28] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [01:43:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.60, 9.20, 8.87 [01:45:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.65, 8.59, 8.69 [01:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:47:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.61, 8.47, 7.87 [01:48:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.32, 9.89, 9.33 [01:49:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.30, 3.31, 2.13 [01:49:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.48, 7.93, 7.74 [01:50:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.64, 3.19 [01:53:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.33, 8.21, 8.44 [01:53:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.21, 9.96, 9.40 [01:55:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.65, 1.79, 1.84 [01:56:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 9.85, 9.67 [01:56:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.94, 2.92, 3.07 [01:58:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.79, 10.65, 9.99 [01:59:07] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.35, 1.65 [02:00:06] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.10, 9.06, 8.69 [02:01:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 9.53, 9.54 [02:02:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.76, 8.78, 8.63 [02:02:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.90, 9.95, 9.91 [02:04:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.97, 7.88, 8.32 [02:05:42] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 13.83, 8.75, 5.37 [02:11:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:11:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.86, 8.36, 8.27 [02:12:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.45, 7.25, 8.34 [02:13:48] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.40, 7.67, 8.01 [02:15:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.46, 1.43 [02:16:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 4.79, 6.78, 8.22 [02:16:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:17:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.50, 1.37, 1.39 [02:19:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.81, 9.00, 8.41 [02:20:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.91, 8.69, 8.68 [02:21:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.65, 2.01, 1.62 [02:21:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.79, 8.54, 8.29 [02:22:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.46, 9.53, 8.99 [02:23:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.56, 1.50 [02:23:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.39, 8.49, 8.30 [02:24:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.12, 8.92, 8.84 [02:27:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.76, 8.61, 8.45 [02:29:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.93, 8.42, 8.40 [02:30:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.20, 7.60, 8.31 [02:36:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:39:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.00, 4.18, 3.24 [02:43:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.96, 3.51, 3.36 [02:45:59] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:13] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 3.08, 3.21 [02:50:03] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.158 second response time [02:50:18] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 3.082 second response time [02:51:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.40, 8.31, 7.99 [02:53:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.48, 8.74, 7.82 [02:57:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 9.17, 8.17 [02:59:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.18, 8.40, 8.22 [02:59:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.92, 8.46, 8.03 [03:02:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.16, 8.18, 7.39 [03:03:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.37, 8.60, 8.20 [03:05:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.96, 9.17, 8.02 [03:06:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 9.38, 8.73 [03:07:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.96, 8.82, 8.02 [03:08:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.90, 8.97, 8.10 [03:09:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.59, 9.95, 8.83 [03:09:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 8.37, 7.30 [03:11:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.51, 3.63, 3.12 [03:11:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.75, 9.50, 8.80 [03:11:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.77, 8.27, 7.41 [03:12:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 4.38, 7.17, 7.66 [03:13:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.18, 9.53, 8.88 [03:13:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.71, 2.77, 2.86 [03:13:42] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 1.32, 3.55, 7.54 [03:16:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.82, 8.34, 8.14 [03:17:28] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.11, 9.80, 9.18 [03:17:42] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.59, 2.44, 6.17 [03:18:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.34, 9.77, 9.26 [03:22:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 9.10, 8.45 [03:26:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 9.05, 8.41 [03:26:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.58, 8.97, 8.60 [03:28:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.80, 8.93, 9.38 [03:28:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.21, 8.39, 8.25 [03:34:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.30, 6.80, 8.26 [03:34:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.75, 7.84, 8.26 [03:35:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 7.86, 8.44 [03:38:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:40:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.24, 8.31, 7.81 [03:41:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.73, 9.55, 8.87 [03:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:41:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.58, 2.87 [03:42:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.15, 8.05, 7.79 [03:42:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.52, 3.08 [03:43:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.01, 9.22, 8.83 [03:43:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.91, 3.26, 2.83 [03:44:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.07, 9.28, 8.61 [03:44:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 3.27, 3.03 [03:46:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.68, 8.45 [03:46:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.35, 8.22, 8.49 [03:48:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.29, 8.70 [03:49:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.55, 1.27 [03:50:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.22, 8.89, 8.63 [03:51:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.38, 1.24 [03:52:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.03, 8.04, 8.36 [03:54:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:56:36] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [03:56:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404689311981201 seconds [03:59:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:00:30] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.69, 1.18 [04:02:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.90, 1.34, 1.11 [04:10:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.93, 9.62, 7.96 [04:10:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.23, 8.37, 7.53 [04:12:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.91, 8.76, 7.84 [04:12:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 8.53, 7.70 [04:14:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 8.28, 7.77 [04:17:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 8.56, 7.84 [04:18:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.87, 8.38, 7.93 [04:19:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.47, 1.24 [04:20:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 9.31, 8.32 [04:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.34, 1.39, 1.24 [04:26:33] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [04:27:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:28:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.53, 8.31, 8.29 [04:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:50:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.38, 7.97 [04:50:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:54:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.82, 8.16, 8.00 [04:55:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:56:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [04:56:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407411098480225 seconds [05:03:39] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.98, 2.12, 1.56 [05:07:59] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:59] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:13] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:13] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:14] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:17] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:21] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:08:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:09:03] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.87, 6.49, 3.91 [05:09:53] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.021 second response time [05:09:57] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.361 second response time [05:10:08] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.021 second response time [05:10:09] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.361 second response time [05:10:09] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.560 second response time [05:10:16] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.584 second response time [05:10:17] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.340 second response time [05:10:24] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.254 second response time [05:11:00] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.41, 6.43, 4.19 [05:26:34] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [05:28:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:29:45] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:56] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:00] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:03] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:37] PROBLEM - mwtask111 SSH on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:54] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:31:59] PROBLEM - mwtask111 JobRunner Service on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:42] PROBLEM - mwtask111 PowerDNS Recursor on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:45] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:32:47] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:33:11] PROBLEM - mwtask111 ferm_active on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:33:28] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:33:57] RECOVERY - mwtask111 JobRunner Service on mwtask111 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [05:34:02] PROBLEM - mwtask111 conntrack_table_size on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:34:04] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 8.793 second response time [05:34:04] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 2.128 second response time [05:34:08] PROBLEM - mwtask111 php-fpm on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:34:12] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.330 second response time [05:34:20] PROBLEM - mwtask111 Disk Space on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:34:37] RECOVERY - mwtask111 PowerDNS Recursor on mwtask111 is OK: DNS OK: 0.152 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [05:34:44] RECOVERY - mwtask111 SSH on mwtask111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [05:34:45] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:34:47] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [05:34:53] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14566 bytes in 0.308 second response time [05:35:07] RECOVERY - mwtask111 ferm_active on mwtask111 is OK: OK ferm input default policy is set [05:35:24] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [05:35:51] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22303 bytes in 0.432 second response time [05:35:58] RECOVERY - mwtask111 conntrack_table_size on mwtask111 is OK: OK: nf_conntrack is 0 % full [05:36:04] RECOVERY - mwtask111 php-fpm on mwtask111 is OK: PROCS OK: 2 processes with command name 'php-fpm7.4' [05:36:17] RECOVERY - mwtask111 Disk Space on mwtask111 is OK: DISK OK - free space: / 31152 MB (47% inode=90%); [05:50:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:55:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:09:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:19:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:20:40] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [06:20:40] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:20:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.64, 3.01 [06:22:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.18, 3.29, 2.95 [06:26:37] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [06:26:37] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404183626174927 seconds [06:29:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:34:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:52:26] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ysmwikiwiki --username-prefix=w --report 1 /home/reception/Wikipedia-20220223003344_%281%29.xml (START) [06:52:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:55:45] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [06:56:33] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [07:02:07] [02mw-config] 07Reception123 commented on pull request 03#4484: T8846: Update Wikibase.php regarding Benpedia's {{WBREPO}} name - 13https://github.com/miraheze/mw-config/pull/4484#issuecomment-1055091221 [07:02:07] https://meta.miraheze.org/wiki/Template:WBREPO [07:02:11] [02mw-config] 07Reception123 closed pull request 03#4484: T8846: Update Wikibase.php regarding Benpedia's {{WBREPO}} name - 13https://github.com/miraheze/mw-config/pull/4484 [07:02:11] https://meta.miraheze.org/wiki/Template:WBREPO [07:02:12] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/62f0c32e2214...5c1928ef238c [07:02:14] [02miraheze/mw-config] 07ugochimobi 035c1928e - Update Wikibase.php regarding Benpedia's {{WBREPO}} name (#4484) [07:02:14] https://meta.miraheze.org/wiki/Template:WBREPO [07:03:04] [02mw-config] 07RhinosF1 commented on pull request 03#4484: T8846: Update Wikibase.php regarding Benpedia's {{WBREPO}} name - 13https://github.com/miraheze/mw-config/pull/4484#issuecomment-1055091998 [07:03:04] https://meta.miraheze.org/wiki/Template:WBREPO [07:03:20] miraheze/mw-config - Reception123 the build passed. [07:04:13] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=empiredarsoiswiki (END - exit=256) [07:04:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:30] !log MariaDB [empiredarsoiswiki]> ALTER TABLE searchindex MODIFY COLUMN si_title MEDIUMTEXT; [07:06:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:56] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildtextindex.php --wiki=empiredarsoiswiki (END - exit=0) [07:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=ysmwikiwiki --username-prefix=w --report 1 /home/reception/Wikipedia-20220223003344_%281%29.xml (END - exit=0) [07:16:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:17:59] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=akb48wiki --username-prefix=stage48 --no-updates --report 1 /home/reception/Wiki48-20220224184049.xml (START) [07:18:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:27] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=akb48wiki --username-prefix=stage48 --no-updates --report 1 /home/reception/Wiki48-20220224184049.xml (END - exit=0) [07:18:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:03] !log [@test101] starting deploy of {'config': True} to all [07:19:04] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [07:19:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:19:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:42] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.56, 1.28 [07:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.12, 1.36, 1.24 [07:23:29] !log [@mwtask111] starting deploy of {'config': True} to all [07:23:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.44, 2.64 [07:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:23:42] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 13s [07:23:45] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:24:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 3.50, 2.70 [07:26:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.07, 2.63 [07:27:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.30, 3.03, 2.67 [07:33:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:38:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:41:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:45:28] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.40, 1.14, 1.81 [07:46:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:47:28] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.26, 0.82, 1.61 [07:49:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:54:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:19:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.77, 1.45 [08:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.53, 1.40 [08:55:13] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jeremyboyd.org' expires in 15 day(s) (Thu 17 Mar 2022 08:47:56 GMT +0000). [08:57:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed3810d05713...6c8cb26e9eac [08:57:33] [02miraheze/ssl] 07MirahezeSSLBot 036c8cb26 - Bot: Update SSL cert for wiki.jeremyboyd.org [09:11:11] PROBLEM - lakehub.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lakehub.ga' expires in 15 day(s) (Thu 17 Mar 2022 08:50:47 GMT +0000). [09:15:03] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sarovia.graalmilitary.com' expires in 15 day(s) (Thu 17 Mar 2022 08:49:22 GMT +0000). [09:17:07] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6c8cb26e9eac...02d369742fed [09:17:09] [02miraheze/ssl] 07MirahezeSSLBot 0302d3697 - Bot: Update SSL cert for lakehub.ga [09:17:10] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanon.polandballwiki.com' expires in 15 day(s) (Thu 17 Mar 2022 08:48:39 GMT +0000). [09:19:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [09:19:04] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:21:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/02d369742fed...86bc14ebbed3 [09:21:57]